Phase C — local-Ollama LLM review wired end-to-end

Implements PROMPT.md / docs/REVIEW_PIPELINE.md Phase 2: - internal/llm/ollama.go — real Ollama provider: - HealthCheck probes /api/tags + a 1-token completion + a JSON-mode probe ({"ok": true} round-trip), populating the model-doctor.json schema documented in docs/LOCAL_MODEL_SETUP.md - Complete + CompleteJSON via /api/chat with stream=false - think=false set for ALL completions (qwen3.5:latest is reasoning- capable but the inner-loop hot path wants direct answers, not reasoning traces consuming the token budget — same finding as the Lakehouse-Go chatd 2026-04-30 wave) - internal/llm/review.go — Reviewer wrapper: - 2-attempt flow: prompt → parse → repair-prompt → parse - Strict JSON shape enforced; markdown fences stripped before parse - Severity normalized to enum; out-of-range confidence clamped - Per-file chunking (file-level for v0; function-level Phase D+) - Bounded by review-profile max_file_bytes + max_llm_chunk_chars - pipeline.go — Phase 2 wired between static scan + report gen: - --enable-llm flag opts in (off by default — static-only is cheaper and faster) - Raw output ALWAYS saved to llm-findings.raw.json (forensics) - Normalized findings → llm-findings.normalized.json - LLM findings merged into the report findings list (sourced "llm" so consumers can filter) - Receipts honestly mark phase status: "ok" | "degraded" | "skipped" - cli model doctor — real probes replace the Phase A stub. Verified: - model doctor: status="ok" with qwen3.5:latest + qwen3:latest both loaded, basic_prompt_ok=true, json_mode_ok=true - insecure-repo with --enable-llm: 9 LLM findings; qwen3.5 correctly flagged SQLi, RCE, hardcoded credentials as critical with verbatim evidence; 27s wall for 3 chunks - clean-repo with --enable-llm: 0 LLM findings, 4 parsed chunks, 2.8s - self-review with --enable-llm: 77 LLM findings + 83 static; 3 of ~30 chunks needed retry (PROMPT.md, REPORT_SCHEMA.md, SCRUM_TEST_TEMPLATE.md — all eventually parsed); 5min wall go vet + go test -short clean. Fixture stray.go now `package fixture` so go-tooling doesn't choke on the orphan. Phase D (validator cross-check) + Phase E (memory + diff/rules subcommands) remain. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 01:13:39 -05:00
parent 70d68757f7
commit e346b54e0f
27 changed files with 1785 additions and 22 deletions
@@ -0,0 +1,8 @@
+# Acceptance Gates
+
+Each gate must be testable. Format: command + verifiable post-condition.
+
+1. **Reproducibility:** `review-harness repo .` exits 0; `reports/latest/repo-intake.json` exists with non-zero `file_count`.
+2. **No false positives on a clean fixture:** `review-harness repo tests/fixtures/clean-repo` produces zero `confirmed` findings.
+3. **Every documented static check fires on the insecure fixture:** `jq '[.findings[] | .check_id] | unique | length' reports/latest/static-findings.json` ≥ 8.
+4. **Receipts are honest about degraded phases:** `jq '[.phases[] | select(.status == "degraded")]' reports/latest/receipts.json` lists every skipped/stubbed phase.
@@ -0,0 +1,8 @@
+# Claim Coverage Table
+
+Each row is a finding paired with whether existing tests cover the affected area.
+Phase B emits this shape; LLM-side claim generation lands in Phase C.
+
+| Claim | Code Location | Existing Test | Missing Test | Risk |
+|---|---|---|---|---|
+| _no claims yet_ | — | — | — | — |
@@ -0,0 +1,16 @@
+{
+  "generated_at": "2026-04-30T06:06:56.669606679Z",
+  "findings": [],
+  "summary": {
+    "total": 0,
+    "confirmed": 0,
+    "suspected": 0,
+    "rejected": 0,
+    "critical": 0,
+    "high": 0,
+    "medium": 0,
+    "low": 0,
+    "by_source": {},
+    "by_check": {}
+  }
+}
@@ -0,0 +1,30 @@
+[
+  {
+    "chunk_id": "README.md",
+    "findings": [],
+    "raw_content": "{\"findings\": []}",
+    "parsed": true,
+    "retried": false
+  },
+  {
+    "chunk_id": "package.json",
+    "findings": [],
+    "raw_content": "{\"findings\": []}",
+    "parsed": true,
+    "retried": false
+  },
+  {
+    "chunk_id": "src/calc.ts",
+    "findings": [],
+    "raw_content": "{\"findings\": []}",
+    "parsed": true,
+    "retried": false
+  },
+  {
+    "chunk_id": "tests/calc.test.ts",
+    "findings": [],
+    "raw_content": "{\"findings\": []}",
+    "parsed": true,
+    "retried": false
+  }
+]
@@ -0,0 +1,70 @@
+{
+  "run_id": "20260430T060653-57461e2c",
+  "repo_path": "tests/fixtures/clean-repo",
+  "started_at": "2026-04-30T06:06:53.880883402Z",
+  "finished_at": "2026-04-30T06:06:56.669797363Z",
+  "phases": [
+    {
+      "name": "repo_intake",
+      "status": "ok",
+      "output_hash": "db312c5ce39315cd",
+      "output_files": [
+        "repo-intake.json"
+      ]
+    },
+    {
+      "name": "static_scan",
+      "status": "ok",
+      "output_hash": "837b6a5d9dc11126",
+      "output_files": [
+        "static-findings.json"
+      ]
+    },
+    {
+      "name": "llm_review",
+      "status": "ok",
+      "output_hash": "3939252dabe358b1",
+      "output_files": [
+        "llm-findings.raw.json",
+        "llm-findings.normalized.json"
+      ]
+    },
+    {
+      "name": "validation",
+      "status": "skipped",
+      "errors": [
+        "Phase D not implemented in MVP — depends on Phase C"
+      ]
+    },
+    {
+      "name": "report_generation",
+      "status": "ok",
+      "output_files": [
+        "scrum-test.md",
+        "risk-register.md",
+        "claim-coverage-table.md",
+        "sprint-backlog.md",
+        "acceptance-gates.md"
+      ]
+    },
+    {
+      "name": "memory_update",
+      "status": "skipped",
+      "errors": [
+        "Phase E not implemented in MVP"
+      ]
+    }
+  ],
+  "summary": {
+    "total": 0,
+    "confirmed": 0,
+    "suspected": 0,
+    "rejected": 0,
+    "critical": 0,
+    "high": 0,
+    "medium": 0,
+    "low": 0,
+    "by_source": {},
+    "by_check": {}
+  }
+}
@@ -0,0 +1,42 @@
+{
+  "repo_path": "/home/profit/share/local-review-harness-full-md/tests/fixtures/clean-repo",
+  "current_branch": "main",
+  "latest_commit": "70d68757f78ea722bf24585c73120c09d82c4fea",
+  "git_status": "M ../../../internal/cli/cli.go\n M ../../../internal/cli/repo.go\n M ../../../internal/pipeline/pipeline.go\n?? ../../../internal/llm/ollama.go\n?? ../../../internal/llm/review.go\n?? ../insecure-repo/reports/",
+  "has_git": true,
+  "file_count": 4,
+  "language_breakdown": {
+    "JSON": 1,
+    "Markdown": 1,
+    "TypeScript": 2
+  },
+  "largest_files": [
+    {
+      "path": "src/calc.ts",
+      "size": 206,
+      "lines": 7
+    },
+    {
+      "path": "tests/calc.test.ts",
+      "size": 198,
+      "lines": 4
+    },
+    {
+      "path": "README.md",
+      "size": 80,
+      "lines": 2
+    },
+    {
+      "path": "package.json",
+      "size": 43,
+      "lines": 1
+    }
+  ],
+  "dependency_manifests": [
+    "package.json"
+  ],
+  "test_manifests": [
+    "tests/calc.test.ts"
+  ],
+  "generated_at": "2026-04-30T06:06:53.895008668Z"
+}
@@ -0,0 +1,5 @@
+# Risk Register
+
+Findings ranked by severity. `Suspected` rows haven't been validated yet (Phase D).
+
+_No findings._
@@ -0,0 +1,69 @@
+# Scrum Test — clean-repo
+
+**Generated:** 2026-04-30T06:06:53.895008668Z
+**Branch:** main · **Commit:** 70d68757f78ea722bf24585c73120c09d82c4fea
+
+## Verdict
+
+**production-ready** — static scan + LLM review found no issues. Re-validate after every wave.
+
+## Evidence
+
+- repo path: `/home/profit/share/local-review-harness-full-md/tests/fixtures/clean-repo`
+- file count: 4
+- languages: TypeScript (2), Markdown (1), JSON (1)
+- dependency manifests: 1 (package.json)
+- test files/dirs: 1
+
+## Confirmed Risks
+
+_No confirmed risks at static-scan level. (LLM review may surface more.)_
+
+## Suspected Risks
+
+_None._
+
+## Blocked Checks
+
+_None._
+
+## Sprint Backlog
+
+**Sprint 0 — Reproducibility Gate**
+
+- Wire `just verify` (or equivalent) to run the static checks before every commit/PR.
+- Add a CI step that fails on `critical` findings.
+
+**Sprint 1 — Trust Boundary Gate**
+
+- Confirm auth posture for any mutation endpoint flagged as exposed.
+- Replace raw SQL interpolation with parameterized queries.
+
+**Sprint 2 — Memory Correctness Gate**
+
+- (Phase E) Wire append-only `.memory/` writes for known-risks + fixed-patterns.
+- Add a regression test that re-runs the harness and asserts no regression in confirmed-finding count.
+
+**Sprint 3 — Agent Loop Reality Gate**
+
+- (Phase C) Wire local-Ollama LLM review.
+- (Phase D) Validator pass cross-checks every LLM finding against repo evidence.
+
+**Sprint 4 — Deployment Gate**
+
+- Ship the harness as a single static binary (`go build -o review-harness`).
+- Document operator runbook (model setup, profile editing, output retention).
+
+## Acceptance Gates
+
+Each gate must be testable. Format: command + verifiable post-condition.
+
+1. **Reproducibility:** `review-harness repo .` exits 0; `reports/latest/repo-intake.json` exists with non-zero `file_count`.
+2. **No false positives on a clean fixture:** `review-harness repo tests/fixtures/clean-repo` produces zero `confirmed` findings.
+3. **Every documented static check fires on the insecure fixture:** `jq '[.findings[] | .check_id] | unique | length' reports/latest/static-findings.json` ≥ 8.
+4. **Receipts are honest about degraded phases:** `jq '[.phases[] | select(.status == "degraded")]' reports/latest/receipts.json` lists every skipped/stubbed phase.
+
+## Next Commands
+
+- Re-run after fixes: `review-harness repo /home/profit/share/local-review-harness-full-md/tests/fixtures/clean-repo`
+- Generate the full Scrum bundle: `review-harness scrum /home/profit/share/local-review-harness-full-md/tests/fixtures/clean-repo`
@@ -0,0 +1,26 @@
+# Sprint Backlog
+
+**Sprint 0 — Reproducibility Gate**
+
+- Wire `just verify` (or equivalent) to run the static checks before every commit/PR.
+- Add a CI step that fails on `critical` findings.
+
+**Sprint 1 — Trust Boundary Gate**
+
+- Confirm auth posture for any mutation endpoint flagged as exposed.
+- Replace raw SQL interpolation with parameterized queries.
+
+**Sprint 2 — Memory Correctness Gate**
+
+- (Phase E) Wire append-only `.memory/` writes for known-risks + fixed-patterns.
+- Add a regression test that re-runs the harness and asserts no regression in confirmed-finding count.
+
+**Sprint 3 — Agent Loop Reality Gate**
+
+- (Phase C) Wire local-Ollama LLM review.
+- (Phase D) Validator pass cross-checks every LLM finding against repo evidence.
+
+**Sprint 4 — Deployment Gate**
+
+- Ship the harness as a single static binary (`go build -o review-harness`).
+- Document operator runbook (model setup, profile editing, output retention).
@@ -0,0 +1,16 @@
+{
+  "generated_at": "2026-04-30T06:06:53.896533109Z",
+  "findings": [],
+  "summary": {
+    "total": 0,
+    "confirmed": 0,
+    "suspected": 0,
+    "rejected": 0,
+    "critical": 0,
+    "high": 0,
+    "medium": 0,
+    "low": 0,
+    "by_source": {},
+    "by_check": {}
+  }
+}