auditor: fix two false-positive classes from cloud inference
lakehouse/auditor 1 blocking issue: cloud: claim not backed — "the proven escalation ladder with learning context, collects"

Observed on PR #8 audit (de11ac4): 7 warn findings, all from the
cloud inference check. Investigation showed two distinct bug classes
that weren't "ship bad code", they were "auditor misreads the diff":

1. Cloud flagged "X not defined in this diff / missing implementation"
   for symbols like `tailJsonl` and `stubFinding` that ARE defined —
   just not in the added lines of this diff. Fix: extract candidate
   symbols from the cloud's gap summary, grep the repo for their
   definitions (function/const/let/def/class/struct/enum/trait/fn).
   If every named symbol resolves, drop the finding; if some do,
   demote to info with the resolution in evidence.

2. Cloud flagged runtime metrics like "58 cloud calls, 306s
   end-to-end" as unbacked claims. These are empirical outputs
   from running the test, not things a static diff can prove.
   Fix: claim_parser now has an `empirical` strength class
   matching iteration counts, cloud-call counts, duration metrics,
   attempt counts, tier-count phrases. Inference drops empirical
   claims from its cloud prompt (verifiable[] subset only) and
   claim-index mapping uses verifiable[] so cloud responses still
   line up.

Added `claims_empirical` to audit metrics so the verdict is
introspectable: how many claims WERE runtime-only vs how many
are diff-verifiable?

Verified: unit tests confirm empirical classification on 5
sample commit messages; symbol resolver found both false-positive
symbols (tailJsonl + stubFinding) and correctly skipped a known-
fake symbol.
This commit is contained in:
profit
2026-04-22 21:40:03 -05:00
parent de11ac4018
commit f4be27a879
4 changed files with 159 additions and 7 deletions
+8 -1
View File
@@ -18,7 +18,14 @@ export interface Claim {
// Heuristic rating of how strong the claim is. "green+tested"
// is strong; "should work" is weak. Drives sensitivity — stronger
// claims get harder-blocked on weak evidence.
strength: "weak" | "moderate" | "strong";
//
// "empirical" is a separate class: runtime measurements like
// "N cloud calls" / "306s end-to-end" / "accepted on attempt N".
// These cannot be verified from a static diff — only from the test
// output that produced them. Inference skips diff-verification for
// empirical claims; they become info-level context unless a future
// runtime_evidence check contradicts them.
strength: "weak" | "moderate" | "strong" | "empirical";
}
export interface Finding {