Phase D — validator cross-checks LLM findings + 2 close-out fixes
Implements PROMPT.md / docs/REVIEW_PIPELINE.md Phase 3:
"AI may suggest. Code validates."
internal/validators/validate.go — 3 hard checks per the
"Reject A Finding If" list:
- file does not exist (with path-traversal guard against the LLM
hallucinating ../../../etc/passwd)
- cited evidence does not appear in the file (verbatim or
trim-line-by-line — models often re-indent quotes when quoting code)
- line hint exceeds file line count
3 soft checks documented as open (claim semantics, suggested-fix
relevance, invented tests/commands — all need another LLM pass).
internal/validators/validate_test.go — 9 tests including:
- TestValidate_RejectsNonexistentFile (gate D1)
- TestValidate_RejectsEvidenceNotInFile
- TestValidate_RejectsLineHintBeyondFile
- TestValidate_AcceptsRealFinding
- TestValidate_AcceptsEvidenceWithDifferentLeadingWhitespace
- TestValidate_RejectsEmptyEvidence
- TestValidate_PassesThroughStaticFindings
- TestValidate_RejectsPathEscapingRepo (path-traversal protection)
- TestValidate_AcceptsRelativeRepoPath (the regression — see below)
Pipeline phase 3 wired between LLM review (Phase C) and report gen
(Phase 4). validated-findings.json contains the confirmed set;
rejected-findings.json contains rejects with per-finding reason +
detail. Receipt phase entry honest about output files + status.
=== Bug J caught ===
First Phase D run rejected EVERY real LLM finding as file_not_found
because the path-traversal check compared a relative joined path
(`tests/fixtures/insecure-repo/src/handler.go`) against an absolute
repoAbs (`/home/profit/share/.../insecure-repo`), so HasPrefix
always returned false. Both sides now resolved via filepath.Abs
before comparison. Regression test
TestValidate_AcceptsRelativeRepoPath locks this in — runs the
validator against a relative repo path AND a relative chdir, the
exact shape that hit the bug.
J's framing was honest: "I don't know what the problem is, but you
know what we're trying to accomplish." The fix-it-yourself signal
let me trace through the rejection details + see the smoking gun
in the detail string ("escapes repo root"). Without that prompt the
9 false rejections might have looked like real LLM bugs.
=== 2 close-out fixes ===
1. .gitignore: changed `/reports/latest/` → `**/reports/latest/`
(and same for `run-*`). Phase C committed 22 generated files
from `tests/fixtures/*/reports/latest/` because the original
pattern was anchored at the harness root only. Existing tracked
files removed via git rm --cached; new pattern keeps fixture
reports out of version control going forward.
2. pipeline.cleanOutputDir: pipeline now deletes the bounded list
of known per-run files at the start of each run. Before this,
a prior run's rejected-findings.json could linger when the
current run had no rejections — confused J during the bug hunt
above. cleanOutputDir is bounded (deletes only files we emit)
so operator-owned adjacent files stay.
Verified end-to-end: insecure-repo + --enable-llm → 25 confirmed
findings (16 static + 9 LLM), 0 rejected.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude (review-harness setup)
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
e346b54e0f
commit
4dc53c5798
@@ -1,8 +0,0 @@
|
||||
# Acceptance Gates
|
||||
|
||||
Each gate must be testable. Format: command + verifiable post-condition.
|
||||
|
||||
1. **Reproducibility:** `review-harness repo .` exits 0; `reports/latest/repo-intake.json` exists with non-zero `file_count`.
|
||||
2. **No false positives on a clean fixture:** `review-harness repo tests/fixtures/clean-repo` produces zero `confirmed` findings.
|
||||
3. **Every documented static check fires on the insecure fixture:** `jq '[.findings[] | .check_id] | unique | length' reports/latest/static-findings.json` ≥ 8.
|
||||
4. **Receipts are honest about degraded phases:** `jq '[.phases[] | select(.status == "degraded")]' reports/latest/receipts.json` lists every skipped/stubbed phase.
|
||||
@@ -1,8 +0,0 @@
|
||||
# Claim Coverage Table
|
||||
|
||||
Each row is a finding paired with whether existing tests cover the affected area.
|
||||
Phase B emits this shape; LLM-side claim generation lands in Phase C.
|
||||
|
||||
| Claim | Code Location | Existing Test | Missing Test | Risk |
|
||||
|---|---|---|---|---|
|
||||
| _no claims yet_ | — | — | — | — |
|
||||
@@ -1,16 +0,0 @@
|
||||
{
|
||||
"generated_at": "2026-04-30T06:06:56.669606679Z",
|
||||
"findings": [],
|
||||
"summary": {
|
||||
"total": 0,
|
||||
"confirmed": 0,
|
||||
"suspected": 0,
|
||||
"rejected": 0,
|
||||
"critical": 0,
|
||||
"high": 0,
|
||||
"medium": 0,
|
||||
"low": 0,
|
||||
"by_source": {},
|
||||
"by_check": {}
|
||||
}
|
||||
}
|
||||
@@ -1,30 +0,0 @@
|
||||
[
|
||||
{
|
||||
"chunk_id": "README.md",
|
||||
"findings": [],
|
||||
"raw_content": "{\"findings\": []}",
|
||||
"parsed": true,
|
||||
"retried": false
|
||||
},
|
||||
{
|
||||
"chunk_id": "package.json",
|
||||
"findings": [],
|
||||
"raw_content": "{\"findings\": []}",
|
||||
"parsed": true,
|
||||
"retried": false
|
||||
},
|
||||
{
|
||||
"chunk_id": "src/calc.ts",
|
||||
"findings": [],
|
||||
"raw_content": "{\"findings\": []}",
|
||||
"parsed": true,
|
||||
"retried": false
|
||||
},
|
||||
{
|
||||
"chunk_id": "tests/calc.test.ts",
|
||||
"findings": [],
|
||||
"raw_content": "{\"findings\": []}",
|
||||
"parsed": true,
|
||||
"retried": false
|
||||
}
|
||||
]
|
||||
@@ -1,70 +0,0 @@
|
||||
{
|
||||
"run_id": "20260430T060653-57461e2c",
|
||||
"repo_path": "tests/fixtures/clean-repo",
|
||||
"started_at": "2026-04-30T06:06:53.880883402Z",
|
||||
"finished_at": "2026-04-30T06:06:56.669797363Z",
|
||||
"phases": [
|
||||
{
|
||||
"name": "repo_intake",
|
||||
"status": "ok",
|
||||
"output_hash": "db312c5ce39315cd",
|
||||
"output_files": [
|
||||
"repo-intake.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "static_scan",
|
||||
"status": "ok",
|
||||
"output_hash": "837b6a5d9dc11126",
|
||||
"output_files": [
|
||||
"static-findings.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "llm_review",
|
||||
"status": "ok",
|
||||
"output_hash": "3939252dabe358b1",
|
||||
"output_files": [
|
||||
"llm-findings.raw.json",
|
||||
"llm-findings.normalized.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "validation",
|
||||
"status": "skipped",
|
||||
"errors": [
|
||||
"Phase D not implemented in MVP — depends on Phase C"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "report_generation",
|
||||
"status": "ok",
|
||||
"output_files": [
|
||||
"scrum-test.md",
|
||||
"risk-register.md",
|
||||
"claim-coverage-table.md",
|
||||
"sprint-backlog.md",
|
||||
"acceptance-gates.md"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "memory_update",
|
||||
"status": "skipped",
|
||||
"errors": [
|
||||
"Phase E not implemented in MVP"
|
||||
]
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"total": 0,
|
||||
"confirmed": 0,
|
||||
"suspected": 0,
|
||||
"rejected": 0,
|
||||
"critical": 0,
|
||||
"high": 0,
|
||||
"medium": 0,
|
||||
"low": 0,
|
||||
"by_source": {},
|
||||
"by_check": {}
|
||||
}
|
||||
}
|
||||
@@ -1,42 +0,0 @@
|
||||
{
|
||||
"repo_path": "/home/profit/share/local-review-harness-full-md/tests/fixtures/clean-repo",
|
||||
"current_branch": "main",
|
||||
"latest_commit": "70d68757f78ea722bf24585c73120c09d82c4fea",
|
||||
"git_status": "M ../../../internal/cli/cli.go\n M ../../../internal/cli/repo.go\n M ../../../internal/pipeline/pipeline.go\n?? ../../../internal/llm/ollama.go\n?? ../../../internal/llm/review.go\n?? ../insecure-repo/reports/",
|
||||
"has_git": true,
|
||||
"file_count": 4,
|
||||
"language_breakdown": {
|
||||
"JSON": 1,
|
||||
"Markdown": 1,
|
||||
"TypeScript": 2
|
||||
},
|
||||
"largest_files": [
|
||||
{
|
||||
"path": "src/calc.ts",
|
||||
"size": 206,
|
||||
"lines": 7
|
||||
},
|
||||
{
|
||||
"path": "tests/calc.test.ts",
|
||||
"size": 198,
|
||||
"lines": 4
|
||||
},
|
||||
{
|
||||
"path": "README.md",
|
||||
"size": 80,
|
||||
"lines": 2
|
||||
},
|
||||
{
|
||||
"path": "package.json",
|
||||
"size": 43,
|
||||
"lines": 1
|
||||
}
|
||||
],
|
||||
"dependency_manifests": [
|
||||
"package.json"
|
||||
],
|
||||
"test_manifests": [
|
||||
"tests/calc.test.ts"
|
||||
],
|
||||
"generated_at": "2026-04-30T06:06:53.895008668Z"
|
||||
}
|
||||
@@ -1,5 +0,0 @@
|
||||
# Risk Register
|
||||
|
||||
Findings ranked by severity. `Suspected` rows haven't been validated yet (Phase D).
|
||||
|
||||
_No findings._
|
||||
@@ -1,69 +0,0 @@
|
||||
# Scrum Test — clean-repo
|
||||
|
||||
**Generated:** 2026-04-30T06:06:53.895008668Z
|
||||
**Branch:** main · **Commit:** 70d68757f78ea722bf24585c73120c09d82c4fea
|
||||
|
||||
## Verdict
|
||||
|
||||
**production-ready** — static scan + LLM review found no issues. Re-validate after every wave.
|
||||
|
||||
## Evidence
|
||||
|
||||
- repo path: `/home/profit/share/local-review-harness-full-md/tests/fixtures/clean-repo`
|
||||
- file count: 4
|
||||
- languages: TypeScript (2), Markdown (1), JSON (1)
|
||||
- dependency manifests: 1 (package.json)
|
||||
- test files/dirs: 1
|
||||
|
||||
## Confirmed Risks
|
||||
|
||||
_No confirmed risks at static-scan level. (LLM review may surface more.)_
|
||||
|
||||
## Suspected Risks
|
||||
|
||||
_None._
|
||||
|
||||
## Blocked Checks
|
||||
|
||||
_None._
|
||||
|
||||
## Sprint Backlog
|
||||
|
||||
**Sprint 0 — Reproducibility Gate**
|
||||
|
||||
- Wire `just verify` (or equivalent) to run the static checks before every commit/PR.
|
||||
- Add a CI step that fails on `critical` findings.
|
||||
|
||||
**Sprint 1 — Trust Boundary Gate**
|
||||
|
||||
- Confirm auth posture for any mutation endpoint flagged as exposed.
|
||||
- Replace raw SQL interpolation with parameterized queries.
|
||||
|
||||
**Sprint 2 — Memory Correctness Gate**
|
||||
|
||||
- (Phase E) Wire append-only `.memory/` writes for known-risks + fixed-patterns.
|
||||
- Add a regression test that re-runs the harness and asserts no regression in confirmed-finding count.
|
||||
|
||||
**Sprint 3 — Agent Loop Reality Gate**
|
||||
|
||||
- (Phase C) Wire local-Ollama LLM review.
|
||||
- (Phase D) Validator pass cross-checks every LLM finding against repo evidence.
|
||||
|
||||
**Sprint 4 — Deployment Gate**
|
||||
|
||||
- Ship the harness as a single static binary (`go build -o review-harness`).
|
||||
- Document operator runbook (model setup, profile editing, output retention).
|
||||
|
||||
## Acceptance Gates
|
||||
|
||||
Each gate must be testable. Format: command + verifiable post-condition.
|
||||
|
||||
1. **Reproducibility:** `review-harness repo .` exits 0; `reports/latest/repo-intake.json` exists with non-zero `file_count`.
|
||||
2. **No false positives on a clean fixture:** `review-harness repo tests/fixtures/clean-repo` produces zero `confirmed` findings.
|
||||
3. **Every documented static check fires on the insecure fixture:** `jq '[.findings[] | .check_id] | unique | length' reports/latest/static-findings.json` ≥ 8.
|
||||
4. **Receipts are honest about degraded phases:** `jq '[.phases[] | select(.status == "degraded")]' reports/latest/receipts.json` lists every skipped/stubbed phase.
|
||||
|
||||
## Next Commands
|
||||
|
||||
- Re-run after fixes: `review-harness repo /home/profit/share/local-review-harness-full-md/tests/fixtures/clean-repo`
|
||||
- Generate the full Scrum bundle: `review-harness scrum /home/profit/share/local-review-harness-full-md/tests/fixtures/clean-repo`
|
||||
@@ -1,26 +0,0 @@
|
||||
# Sprint Backlog
|
||||
|
||||
**Sprint 0 — Reproducibility Gate**
|
||||
|
||||
- Wire `just verify` (or equivalent) to run the static checks before every commit/PR.
|
||||
- Add a CI step that fails on `critical` findings.
|
||||
|
||||
**Sprint 1 — Trust Boundary Gate**
|
||||
|
||||
- Confirm auth posture for any mutation endpoint flagged as exposed.
|
||||
- Replace raw SQL interpolation with parameterized queries.
|
||||
|
||||
**Sprint 2 — Memory Correctness Gate**
|
||||
|
||||
- (Phase E) Wire append-only `.memory/` writes for known-risks + fixed-patterns.
|
||||
- Add a regression test that re-runs the harness and asserts no regression in confirmed-finding count.
|
||||
|
||||
**Sprint 3 — Agent Loop Reality Gate**
|
||||
|
||||
- (Phase C) Wire local-Ollama LLM review.
|
||||
- (Phase D) Validator pass cross-checks every LLM finding against repo evidence.
|
||||
|
||||
**Sprint 4 — Deployment Gate**
|
||||
|
||||
- Ship the harness as a single static binary (`go build -o review-harness`).
|
||||
- Document operator runbook (model setup, profile editing, output retention).
|
||||
@@ -1,16 +0,0 @@
|
||||
{
|
||||
"generated_at": "2026-04-30T06:06:53.896533109Z",
|
||||
"findings": [],
|
||||
"summary": {
|
||||
"total": 0,
|
||||
"confirmed": 0,
|
||||
"suspected": 0,
|
||||
"rejected": 0,
|
||||
"critical": 0,
|
||||
"high": 0,
|
||||
"medium": 0,
|
||||
"low": 0,
|
||||
"by_source": {},
|
||||
"by_check": {}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user