cohesion/python-inventory-and-integration
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 profitandClaude Opus 4.7
|
6e39d8778f |
Cohesion: Python inventory + integration plan + Phase A verdict indexing
lakehouse/auditor all checks passed (3 findings, all info)
Three artifacts in one PR:
1. docs/PYTHON_INVENTORY.md — every .py file in the repo classified:
Production (sidecar routers + 3 systemd services), Documented
(kb_measure, kb_staffer_report), Manual (one-off tools), Dead
(sidecar/sidecar/lab_ui.py + pipeline_lab.py are genuinely
not imported anywhere).
2. docs/COHESION_INTEGRATION_PLAN.md — the "smarter DB" loop J
called out as missing. Six phases A-F. Phase A ships here; B-F
are named + sequenced for follow-up PRs. Each phase adds ONE
wire of the loop; no single PR does them all.
3. Phase A wire (auditor verdicts → observer + KB):
- auditor/audit.ts: after assembleVerdict, fire-and-forget POST
to :3800/event with source="auditor" AND append to
data/_kb/outcomes.jsonl with kind="audit". Errors log + drop
— the verdict is still on disk at _auditor/verdicts/.
- mcp-server/observer.ts: extend source union to include
"auditor" | "bot" (was "mcp" | "scenario" only, which silently
coerced my first auditor POST to source="scenario"). Accept
body.ok OR body.success. Accept body.audit_duration_ms as a
fallback for duration_ms. Uses body.one_liner as
output_summary when set.
Live-verified after observer restart:
re-audit PR #6 → verdict=request_changes, 4 findings (1 warn)
observer: by_source={'auditor': 1} (previously coerced to 'scenario')
_kb/outcomes.jsonl tail: kind=audit sig=pr6-7fe47bab
pr=6 overall=request_changes
The shape of the loop is now visible to downstream consumers. Phase
B (auditor's kb_query check reads these audit rows for history)
lands in a follow-up PR. Phase C-F similar.
NOT in this PR:
- Actually deleting lab_ui.py + pipeline_lab.py (operator decision,
called out in the inventory doc)
- Cleaning up the 5 overlapping Python scripts (same)
- Phases B-F of the cohesion plan (separate PRs per wire)
- Integration test that asserts "smarter DB" across runs (Phase F)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
profit
|
c33c1bcbc5 |
Auditor: poller + live end-to-end proof
lakehouse/auditor all checks passed (4 findings, all info)
auditor/index.ts (task #9) — the top-level poller. 90s interval, dedupes by head SHA via data/_auditor/state.json, supports --once for CLI testing. Env gates: LH_AUDITOR_RUN_DYNAMIC=1 to include the hybrid fixture (default off; it mutates live state), LH_AUDITOR_SKIP_INFERENCE=1 for fast runs without cloud calls. Single-shot run proof (task #10): cycle 1: 2 open PRs audit PR #2 |
||
|
profit
|
039ed32411 |
Auditor: KB query check + verdict orchestrator + Gitea poster
lakehouse/auditor all checks passed (4 findings, all info)
auditor/checks/kb_query.ts (task #7) — reads data/_kb/outcomes.jsonl, error_corrections.jsonl, data/_observer/ops.jsonl, data/_bot/cycles/*. Cheap/offline: no model calls, tail-reads only. Fail-rate >30% in recent scenario outcomes → warn; otherwise info. Live-proven: 1 finding emitted against current KB state (69 scenario runs, 27.7% fail rate — below warn threshold). auditor/audit.ts (task #8) — orchestrator. Runs static + dynamic + inference + kb_query in parallel, calls assembleVerdict, persists to data/_auditor/verdicts/, posts to Gitea (commit status + issue comment). AuditOptions supports skip_dynamic/skip_inference/dry_run for iteration. auditor/gitea.ts — added postIssueComment (author can comment on own PR, unlike postReview which self-review-blocks). static.ts — skip BLOCK_PATTERNS scan on auditor/checks/* and auditor/fixtures/* because those files legitimately contain the patterns as regex/string-literal data. WARN/INFO patterns (TODO comments, hardcoded placeholders) still run. Live-proven: dry-run audit of PR #1 after fix went from 13 block findings to 0 from static; 11 warn from inference still fire on real overreach claims. Dry-run audit against PR #1, skip_dynamic=true: verdict: block (BEFORE the static fix) verdict: request_changes (AFTER — inference correctly flagged "tasks 1-9 complete" as not backed; 0 false-positive blocks from static self-match) 42.5s total across checks (mostly cloud inference: 36s) 26 claims, 39KB diff Tasks 5 + 6 + 7 + 8 complete. Remaining: #9 (poller) + #10 (end-to-end proof) + #12 (upsert UPDATE merge fix). |
||
|
profit
|
efc7b5ac44 |
Auditor: dynamic + inference checks
auditor/checks/dynamic.ts — wraps runHybridFixture, maps layer results to Findings. Placeholder-style errors (404/unimplemented/ slice N) → info; other failures → warn. Always emits a summary finding with real numbers (shipped/placeholder phase counts + per- layer latency). Live-tested against current stack: 2 info findings, 0 warnings — all shipped layers actually work. auditor/checks/inference.ts — wraps the run_codereview reviewer pattern from llm_team_ui.py, adapted for claim-vs-diff verification. Calls /v1/chat provider=ollama_cloud model=gpt-oss:120b. Requests strict JSON response with claim_verdicts[] and unflagged_gaps[]. A strong claim marked "not backed" by cloud → BLOCK severity; moderate → warn; weak → info. Cloud-unreachable or unparseable-output → info (never blocks on the reviewer being down). Live-tested against PR #1 (this PR, 20 claims, 39KB diff): - 36.9s round-trip - 7 block + 23 warn + 2 info findings - gpt-oss:120b correctly flagged "Fully-functional auditor (tasks 1-9 complete)" as not-backed (only 6/10 tasks done at that commit) — accurate catch - Some false positives from the original 15KB truncation threshold (cloud missed gitea.ts, flagged "no Gitea client present") - Bumped MAX_DIFF_CHARS from 15000 to 40000 to fit the full PR diff in context; reviewer precision improves accordingly Tasks 5 + 6 completed. Remaining: #7 (KB query), #8 (verdict + Gitea poster), #9 (poller), #10 (end-to-end proof), #12 (upsert UPDATE-drops-doc_refs). |
||
|
profit
|
c5da680add |
Fixture: unique-per-run nonce eliminates state-pollution false positive
After the serde fix (PR #2, fix/upsert-outcome-serde) landed on main, re-running this fixture STILL reported "doc_refs field is empty" — but with a different root cause than the panic. Root cause: pre-fix runs panicked on response serialization but had already added entries to state (panic happened between upsert_entry returning and the handler's serde_json::json! of the response). So state.json was polluted with __auditor_test_worker__ entries from those runs, WITHOUT doc_refs (doc_refs wasn't even wired at the time those state rows were written). The fixture's `find(endorsed_names.includes(TEST_WORKER_NAME))` was picking the oldest polluted entry, not the fresh one. Compounding: discovered a secondary bug while investigating — upsert_entry's UPDATE branch only merges endorsed_names. doc_refs, schema_fingerprint, valid_until on an UPDATE are silently dropped. Filed as task #12, separate PR to follow. Fix in this fixture: use a nonce suffix on both TEST_WORKER_NAME and TEST_OPERATION so every run is guaranteed to hit the ADD path in upsert_entry, sidestepping the UPDATE bug AND eliminating state pollution entirely. Live re-run after this edit: ✓ Phase 38 /v1/chat 449ms, 42 tokens ✓ Phase 40 Langfuse trace 20ms ✓ Phase 45.1 seed + doc_refs 239ms, doc_refs.length=1 persisted ✓ Phase 45.2 bridge diff 2ms, drifted=true ✗ Phase 45.3 drift-check HONEST 404 (endpoint not built) shipped_phases: [38, 40, 45.1, 45.2] (was [38, 40, 45.2]) placeholder: [45.3] (was [45.1, 45.3]) One fewer placeholder — exactly because the serde fix merged on fix/upsert-outcome-serde and the fixture now cleanly exercises the path. The loop is: fixture finds bug → PR fixes bug → fixture re-run confirms fix → one fewer placeholder. |
||
|
profit
|
5bbcaf4c33 |
Fix: layer-2 Langfuse filter used meaningless ternary
Caught by running a side-test through LLM Team's run_codereview
flow (gpt-oss:120b reviewer) against this fixture, 2026-04-22.
BEFORE:
const ourStart = Date.parse(
l1.evidence.match(/tokens=/) ? result.ran_at : result.ran_at
);
// Both branches return result.ran_at — the ternary is meaningless.
// result.ran_at is the fixture start time, NOT the moment we fired
// /v1/chat. Any trace created between fixture-start and chat-fetch
// would false-negative.
AFTER:
const chat_request_sent_ms = Date.now(); // captured before layer 1
// ...
const recent = items.filter(t =>
Date.parse(t.timestamp) >= chat_request_sent_ms
);
Re-ran the fixture against the live stack — layers 1,2,4 still pass
(no regression); layer 2 trace matched at age=2494ms which is within
the chat-to-trace propagation window. Layers 3,5 still fail for the
original unrelated reasons (UpsertOutcome serde panic + Phase 45
slice 3 endpoint not built).
First concrete act-on-finding from a code-checker run. The process
works.
|
||
|
profit
|
9c893fbb8c |
Auditor: hybrid fixture — found a pre-existing bug on first live run
auditor/fixtures/hybrid_38_40_45.ts — the never-before-run hybrid
test. Exercises Phase 38 /v1/chat → Phase 40 Langfuse → Phase 45
slice 1 seed+doc_refs → Phase 45 slice 2 bridge drift → (expected-
fail) Phase 45 slice 3 drift-check endpoint.
auditor/fixtures/cli.ts — standalone runner. Human-readable summary
to stderr, machine-readable JSON to stdout, exit code 0/1/2 for
pass / fail / partial_pass.
Live run results — honest measurements, not hand-waved:
✓ Phase 38 /v1/chat returns 9 visible tokens, 6.7s latency
("docker run is a common Docker command.")
✓ Phase 40 Langfuse trace 18a8a0b7 landed in 2.5s
✗ Phase 45.1 seed endpoint returns empty reply — discovered a
PRE-EXISTING BUG unrelated to doc_refs:
playbook_memory.rs:257 UpsertOutcome has newtype
variants Added(String) and Noop(String) under
#[serde(tag="mode")] — serde panics on serialize.
panicked at crates/vectord/src/service.rs:2323:
Error("cannot serialize tagged newtype variant
UpsertOutcome::Added containing a string")
Reproduced: curl /seed with AND without doc_refs
both get "Empty reply from server" (socket closed
mid-response). This bug has existed since Phase 26
shipped (commit
|
||
|
profit
|
b933334ae2 |
Auditor: static diff check — catches own Phase 45 placeholder
auditor/checks/static.ts — grep-style scan of PR diffs, no AST,
no LLM. High-signal patterns only.
Severity grading:
- BLOCK — unimplemented!(), todo!(), panic!("not implemented"),
throw new Error("not implemented")
- WARN — TODO/FIXME/XXX/HACK in added lines;
new pub struct fields with <2 mentions in the diff
(added but nobody reads it — placeholder state)
- INFO — hardcoded "placeholder"/"dummy"/"foobar"/"changeme"/"xxx"
strings in added lines
Live-proven — the existential test J asked for:
vs PR #1 (scaffold): 0 findings (all scaffold fields cross-
reference within the diff)
vs commit
|
||
|
profit
|
bfe8985233 |
Auditor: claim parser
auditor/claim_parser.ts — reads PR body + commit messages, extracts
ship-claims. Regex-based, intentionally not LLM-driven: the parser's
job is to surface claim substrates, not to judge them (that's the
inference check's job, runs later with cloud model).
Three strength tiers:
- strong — "verified end-to-end", "live-proven", "production-ready",
"phase N shipped", "proven"
- moderate — "shipped", "landed", "green", "passing", "works",
"complete", "done"
- weak — "should work", "expected to", "probably"
Live-proven against PR #1 (this PR): 4 claims extracted from
1 commit (2 strong, 2 moderate). "live-proven" correctly tagged as
strong (it IS a stronger claim than "shipped").
Next: static diff check consumes these claims + the PR diff to find
placeholder patterns — empty fns, TODO, unwired fields, etc.
|
||
|
profitandClaude Opus 4.7
|
f48dd2f20b |
Auditor scaffold: types + Gitea client + policy stub + README
All-Bun sub-agent that watches open PRs on Gitea, reads ship-claims, and hard-blocks merges when the code doesn't back the claim. First commit of N; this is the skeleton. Dynamic/static/inference/kb checks + poller land in follow-up commits on this same branch. - auditor/types.ts — Claim, Finding, Verdict, PrSnapshot shapes - auditor/gitea.ts — minimal API client (listOpenPrs, getPrDiff, postCommitStatus, postReview). Live-proven: returned 0 open PRs against our repo (which IS the current state — every commit today went to main directly, which is the problem this auditor is meant to prevent) - auditor/policy.ts — stub `assembleVerdict` + severity rules. Intentionally conservative defaults: strong claim + zero evidence = block, not warn. - auditor/README.md — how to run + the hard-block mechanism Workflow discipline change: starting with this branch, no more direct pushes to main. Every change lands as a PR. When this auditor is fully built and running, it'll review its own completion PR — the recursive self-test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |