Phase A + B (MVP) — local review harness
Implements the MVP cutline from the planning artifact: - Phase A: skeleton + CLI dispatch + provider interface + stub model doctor - Phase B: scanner + git probe + 12 static analyzers + reporters + pipeline - Phase B fixtures: clean-repo, insecure-repo, degraded-repo 12 static analyzers per PROMPT.md "Suggested Static Checks For MVP": hardcoded_paths, shell_execution, raw_sql_interpolation, broad_cors, secret_patterns, large_files, todo_comments, missing_tests, env_file_committed, unsafe_file_io, exposed_mutation_endpoint, hardcoded_local_ip. Acceptance gates passing: - B1 (intake produces accurate counts) ✓ - B2 (insecure fixture fires ≥8 distinct check_ids — actually 11/12) ✓ - B3 (clean fixture produces 0 confirmed findings — no false positives) ✓ - B4 (scrum mode produces all 6 required markdown + JSON reports) ✓ - B5 (receipts.json marks degraded phases honestly) ✓ - F (self-review on this repo runs without crashing) ✓ — exit 66 (degraded because Phase C LLM review is hardcoded skipped) Phases C (LLM review), D (validation cross-check), E (memory + diff + rules subcommands) deferred per the cutline. The MVP delivers the evidence-first path; LLM is purely additive. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude (review-harness setup)
co-authored by
Claude Opus 4.7
Claude Opus 4.7
commit
f3ee4722a8
Executable
+86
@@ -0,0 +1,86 @@
|
||||
# Local Model Setup
|
||||
|
||||
## Purpose
|
||||
|
||||
The review harness should use local models first.
|
||||
|
||||
The first supported provider is Ollama.
|
||||
|
||||
The design must allow OpenAI-compatible local endpoints later.
|
||||
|
||||
## Default Ollama Profile
|
||||
|
||||
```yaml
|
||||
provider: ollama
|
||||
base_url: http://localhost:11434
|
||||
model: qwen2.5-coder
|
||||
fallback_model: llama3.1
|
||||
timeout_seconds: 120
|
||||
temperature: 0.1
|
||||
```
|
||||
|
||||
## Model Doctor Command
|
||||
|
||||
The harness must provide:
|
||||
|
||||
```bash
|
||||
review-harness model doctor
|
||||
```
|
||||
|
||||
## Doctor Checks
|
||||
|
||||
The doctor command should test:
|
||||
|
||||
- Ollama server availability
|
||||
- configured model availability
|
||||
- fallback model availability
|
||||
- basic prompt response
|
||||
- JSON response reliability
|
||||
- timeout behavior
|
||||
- degraded-mode behavior
|
||||
|
||||
## Required Doctor Output
|
||||
|
||||
```text
|
||||
reports/latest/model-doctor.json
|
||||
```
|
||||
|
||||
## Required JSON Fields
|
||||
|
||||
```json
|
||||
{
|
||||
"provider": "ollama",
|
||||
"base_url": "http://localhost:11434",
|
||||
"primary_model": "",
|
||||
"fallback_model": "",
|
||||
"server_available": false,
|
||||
"primary_model_available": false,
|
||||
"fallback_model_available": false,
|
||||
"basic_prompt_ok": false,
|
||||
"json_mode_ok": false,
|
||||
"timeout_seconds": 120,
|
||||
"status": "ok|degraded|failed",
|
||||
"errors": []
|
||||
}
|
||||
```
|
||||
|
||||
## Provider Interface
|
||||
|
||||
Do not hardcode Ollama into all logic.
|
||||
|
||||
Use a provider interface with these operations:
|
||||
|
||||
```text
|
||||
list_models()
|
||||
complete(prompt, options)
|
||||
complete_json(prompt, schema, options)
|
||||
health_check()
|
||||
```
|
||||
|
||||
## Local Model Rules
|
||||
|
||||
- temperature should default low for review tasks
|
||||
- prompts should request strict JSON where possible
|
||||
- raw model output must be saved for failed parse attempts
|
||||
- invalid model output must never be silently accepted
|
||||
- fallback model usage must be recorded
|
||||
Executable
+150
@@ -0,0 +1,150 @@
|
||||
# Report Schema
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines the expected report and receipt schemas for the local review harness.
|
||||
|
||||
## Finding Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "",
|
||||
"title": "",
|
||||
"severity": "low|medium|high|critical",
|
||||
"status": "confirmed|suspected|rejected|blocked",
|
||||
"file": "",
|
||||
"line_hint": "",
|
||||
"evidence": "",
|
||||
"reason": "",
|
||||
"suggested_fix": "",
|
||||
"source": "static|llm|validator",
|
||||
"confidence": 0.0
|
||||
}
|
||||
```
|
||||
|
||||
## Severity Rules
|
||||
|
||||
### Critical
|
||||
|
||||
Use for:
|
||||
|
||||
- credential exposure
|
||||
- destructive command risk
|
||||
- unauthenticated mutation endpoint
|
||||
- remote code execution risk
|
||||
- data corruption risk
|
||||
|
||||
### High
|
||||
|
||||
Use for:
|
||||
|
||||
- SQL injection risk
|
||||
- broad CORS on sensitive service
|
||||
- fail-open security behavior
|
||||
- unsafe filesystem access
|
||||
- missing validation on critical inputs
|
||||
|
||||
### Medium
|
||||
|
||||
Use for:
|
||||
|
||||
- hardcoded paths
|
||||
- excessive file size
|
||||
- weak error handling
|
||||
- missing tests around important code
|
||||
- fragile environment assumptions
|
||||
|
||||
### Low
|
||||
|
||||
Use for:
|
||||
|
||||
- minor duplication
|
||||
- naming confusion
|
||||
- documentation drift
|
||||
- small maintainability issues
|
||||
|
||||
## Scrum Test Report Sections
|
||||
|
||||
Every Scrum test report must include:
|
||||
|
||||
```text
|
||||
Verdict
|
||||
Evidence
|
||||
Confirmed Risks
|
||||
Suspected Risks
|
||||
Blocked Checks
|
||||
Sprint Backlog
|
||||
Acceptance Gates
|
||||
Next Commands
|
||||
```
|
||||
|
||||
## Risk Register Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"risks": [
|
||||
{
|
||||
"id": "",
|
||||
"title": "",
|
||||
"severity": "",
|
||||
"affected_area": "",
|
||||
"evidence": "",
|
||||
"impact": "",
|
||||
"mitigation": "",
|
||||
"owner": "",
|
||||
"status": "open|mitigated|accepted|blocked"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Receipt Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"run_id": "",
|
||||
"repo_path": "",
|
||||
"started_at": "",
|
||||
"finished_at": "",
|
||||
"phases": [
|
||||
{
|
||||
"name": "",
|
||||
"status": "ok|degraded|failed|skipped",
|
||||
"input_hash": "",
|
||||
"output_hash": "",
|
||||
"output_files": [],
|
||||
"errors": []
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"confirmed_findings": 0,
|
||||
"suspected_findings": 0,
|
||||
"blocked_checks": 0,
|
||||
"critical": 0,
|
||||
"high": 0,
|
||||
"medium": 0,
|
||||
"low": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Claim Coverage Table
|
||||
|
||||
Use this Markdown table:
|
||||
|
||||
```text
|
||||
| Claim | Code Location | Existing Test | Missing Test | Risk |
|
||||
|---|---|---|---|---|
|
||||
```
|
||||
|
||||
## No Fake Evidence Rule
|
||||
|
||||
Reports must not include:
|
||||
|
||||
- invented file paths
|
||||
- invented command output
|
||||
- invented tests
|
||||
- unsupported claims
|
||||
- false pass/fail statements
|
||||
|
||||
If evidence is missing, say missing evidence.
|
||||
Executable
+186
@@ -0,0 +1,186 @@
|
||||
# Review Pipeline Specification
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines the local review harness pipeline.
|
||||
|
||||
The pipeline exists to inspect a repository, collect evidence, identify risks, validate model claims, and generate operational reports without relying on cloud services.
|
||||
|
||||
## Pipeline Overview
|
||||
|
||||
```text
|
||||
Repo Intake
|
||||
-> Static Scan
|
||||
-> Optional LLM Review
|
||||
-> Validation
|
||||
-> Report Generation
|
||||
-> Memory Update
|
||||
```
|
||||
|
||||
## Phase 0: Repo Intake
|
||||
|
||||
### Goal
|
||||
|
||||
Build a factual profile of the target repository.
|
||||
|
||||
### Inputs
|
||||
|
||||
- repository path
|
||||
- git metadata
|
||||
- filesystem metadata
|
||||
- dependency manifests
|
||||
- build files
|
||||
- test files
|
||||
|
||||
### Required Output
|
||||
|
||||
```text
|
||||
reports/latest/repo-intake.json
|
||||
```
|
||||
|
||||
### Required Fields
|
||||
|
||||
```json
|
||||
{
|
||||
"repo_path": "",
|
||||
"current_branch": "",
|
||||
"latest_commit": "",
|
||||
"git_status": "",
|
||||
"file_count": 0,
|
||||
"language_breakdown": {},
|
||||
"largest_files": [],
|
||||
"dependency_manifests": [],
|
||||
"test_manifests": [],
|
||||
"generated_at": ""
|
||||
}
|
||||
```
|
||||
|
||||
## Phase 1: Static Scan
|
||||
|
||||
### Goal
|
||||
|
||||
Find evidence-backed problems without using an LLM.
|
||||
|
||||
### Detection Targets
|
||||
|
||||
- hardcoded absolute paths
|
||||
- unsafe shell execution
|
||||
- raw SQL interpolation
|
||||
- exposed mutation endpoints
|
||||
- broad CORS
|
||||
- unchecked file reads and writes
|
||||
- suspicious secret patterns
|
||||
- large files
|
||||
- TODO, FIXME, HACK comments
|
||||
- missing tests near critical modules
|
||||
|
||||
### Required Output
|
||||
|
||||
```text
|
||||
reports/latest/static-findings.json
|
||||
```
|
||||
|
||||
## Phase 2: LLM Review
|
||||
|
||||
### Goal
|
||||
|
||||
Use a local model to perform higher-level reasoning over bounded evidence chunks.
|
||||
|
||||
### Rules
|
||||
|
||||
- Do not send the entire repository blindly.
|
||||
- Chunk inputs by file, function, or diff boundary.
|
||||
- Require strict JSON output.
|
||||
- Retry invalid JSON once.
|
||||
- Save degraded output if parsing fails.
|
||||
- Never trust model claims without validation.
|
||||
|
||||
### Required Output
|
||||
|
||||
```text
|
||||
reports/latest/llm-findings.raw.json
|
||||
reports/latest/llm-findings.normalized.json
|
||||
```
|
||||
|
||||
## Phase 3: Validation
|
||||
|
||||
### Goal
|
||||
|
||||
Validate every LLM-generated finding against real repository evidence.
|
||||
|
||||
### Reject A Finding If
|
||||
|
||||
- the file does not exist
|
||||
- the cited evidence does not exist
|
||||
- the line hint is impossible
|
||||
- the claim is unsupported
|
||||
- the suggested fix targets unrelated code
|
||||
- the model invents tests, commands, or files
|
||||
|
||||
### Required Output
|
||||
|
||||
```text
|
||||
reports/latest/validated-findings.json
|
||||
```
|
||||
|
||||
## Phase 4: Report Generation
|
||||
|
||||
### Goal
|
||||
|
||||
Produce human-readable and machine-readable reports.
|
||||
|
||||
### Required Markdown Reports
|
||||
|
||||
```text
|
||||
reports/latest/scrum-test.md
|
||||
reports/latest/risk-register.md
|
||||
reports/latest/claim-coverage-table.md
|
||||
reports/latest/sprint-backlog.md
|
||||
reports/latest/acceptance-gates.md
|
||||
```
|
||||
|
||||
### Required JSON Receipt
|
||||
|
||||
```text
|
||||
reports/latest/receipts.json
|
||||
```
|
||||
|
||||
## Phase 5: Memory
|
||||
|
||||
### Goal
|
||||
|
||||
Persist durable review knowledge for future runs.
|
||||
|
||||
### Required Memory Files
|
||||
|
||||
```text
|
||||
.memory/review-rules.md
|
||||
.memory/known-risks.json
|
||||
.memory/fixed-patterns.json
|
||||
.memory/project-profile.json
|
||||
```
|
||||
|
||||
### Memory Rules
|
||||
|
||||
- append-only by default
|
||||
- version every update
|
||||
- never silently overwrite
|
||||
- record source run ID
|
||||
- record evidence file
|
||||
- record confidence level
|
||||
|
||||
## Degraded Mode
|
||||
|
||||
A phase is degraded when it cannot fully run but the pipeline can continue.
|
||||
|
||||
Examples:
|
||||
|
||||
- Ollama unavailable
|
||||
- model returns invalid JSON
|
||||
- repository has no git metadata
|
||||
- dependency manager unavailable
|
||||
- large dataset missing
|
||||
|
||||
Degraded mode must be explicit in reports.
|
||||
|
||||
No silent success.
|
||||
Executable
+73
@@ -0,0 +1,73 @@
|
||||
# Scrum Test Template
|
||||
|
||||
## Verdict
|
||||
|
||||
State whether the repository is:
|
||||
|
||||
- production-ready
|
||||
- prototype-ready
|
||||
- demo-only
|
||||
- blocked
|
||||
|
||||
Do not soften the verdict.
|
||||
|
||||
## Evidence
|
||||
|
||||
List commands run, files inspected, and outputs generated.
|
||||
|
||||
## Confirmed Risks
|
||||
|
||||
Each confirmed risk must include:
|
||||
|
||||
- file path
|
||||
- evidence
|
||||
- severity
|
||||
- impact
|
||||
- suggested fix
|
||||
|
||||
## Suspected Risks
|
||||
|
||||
Each suspected risk must explain what evidence is missing.
|
||||
|
||||
## Blocked Checks
|
||||
|
||||
List checks that could not run.
|
||||
|
||||
For each blocked check, include:
|
||||
|
||||
- reason
|
||||
- dependency
|
||||
- next command
|
||||
- risk of not running it
|
||||
|
||||
## Sprint Backlog
|
||||
|
||||
Use this format:
|
||||
|
||||
```text
|
||||
Sprint 0: Reproducibility Gate
|
||||
Sprint 1: Trust Boundary Gate
|
||||
Sprint 2: Memory Correctness Gate
|
||||
Sprint 3: Agent Loop Reality Gate
|
||||
Sprint 4: Deployment Gate
|
||||
```
|
||||
|
||||
## Acceptance Gates
|
||||
|
||||
Each gate must be testable.
|
||||
|
||||
Bad:
|
||||
|
||||
```text
|
||||
Improve security.
|
||||
```
|
||||
|
||||
Good:
|
||||
|
||||
```text
|
||||
SQL interpolation scanner detects unsafe SELECT/INSERT/UPDATE/DELETE string assembly and emits confirmed findings with file evidence.
|
||||
```
|
||||
|
||||
## Next Commands
|
||||
|
||||
List only commands that can actually be run.
|
||||
Reference in New Issue
Block a user