auditor: fix two false-positive classes from cloud inference
lakehouse/auditor 1 blocking issue: cloud: claim not backed — "the proven escalation ladder with learning context, collects"
lakehouse/auditor 1 blocking issue: cloud: claim not backed — "the proven escalation ladder with learning context, collects"
Observed on PR #8 audit (de11ac4): 7 warn findings, all from the
cloud inference check. Investigation showed two distinct bug classes
that weren't "ship bad code", they were "auditor misreads the diff":
1. Cloud flagged "X not defined in this diff / missing implementation"
for symbols like `tailJsonl` and `stubFinding` that ARE defined —
just not in the added lines of this diff. Fix: extract candidate
symbols from the cloud's gap summary, grep the repo for their
definitions (function/const/let/def/class/struct/enum/trait/fn).
If every named symbol resolves, drop the finding; if some do,
demote to info with the resolution in evidence.
2. Cloud flagged runtime metrics like "58 cloud calls, 306s
end-to-end" as unbacked claims. These are empirical outputs
from running the test, not things a static diff can prove.
Fix: claim_parser now has an `empirical` strength class
matching iteration counts, cloud-call counts, duration metrics,
attempt counts, tier-count phrases. Inference drops empirical
claims from its cloud prompt (verifiable[] subset only) and
claim-index mapping uses verifiable[] so cloud responses still
line up.
Added `claims_empirical` to audit metrics so the verdict is
introspectable: how many claims WERE runtime-only vs how many
are diff-verifiable?
Verified: unit tests confirm empirical classification on 5
sample commit messages; symbol resolver found both false-positive
symbols (tailJsonl + stubFinding) and correctly skipped a known-
fake symbol.
This commit is contained in:
+8
-1
@@ -18,7 +18,14 @@ export interface Claim {
|
||||
// Heuristic rating of how strong the claim is. "green+tested"
|
||||
// is strong; "should work" is weak. Drives sensitivity — stronger
|
||||
// claims get harder-blocked on weak evidence.
|
||||
strength: "weak" | "moderate" | "strong";
|
||||
//
|
||||
// "empirical" is a separate class: runtime measurements like
|
||||
// "N cloud calls" / "306s end-to-end" / "accepted on attempt N".
|
||||
// These cannot be verified from a static diff — only from the test
|
||||
// output that produced them. Inference skips diff-verification for
|
||||
// empirical claims; they become info-level context unless a future
|
||||
// runtime_evidence check contradicts them.
|
||||
strength: "weak" | "moderate" | "strong" | "empirical";
|
||||
}
|
||||
|
||||
export interface Finding {
|
||||
|
||||
Reference in New Issue
Block a user