proof harness Phase A: scaffolding + canary case green
Per docs/TEST_PROOF_SCOPE.md, building the claims-verification tier
above the smoke chain. This commit lays the scaffolding and proves
the orchestrator end-to-end with one canary case (00_health).
What landed:
tests/proof/
README.md how to read a report, layout, modes
claims.yaml 24 claims enumerated (GOLAKE-001..100)
run_proof.sh orchestrator with --mode {contract|integration|performance}
and --no-bootstrap / --regenerate-{rankings,baseline}
lib/
env.sh service URLs, report dir, mode, git context
http.sh curl wrappers writing per-probe JSON + body + headers
assert.sh proof_assert_{eq,ne,contains,lt,gt,status,json_eq} +
proof_skip — each emits one JSONL record per call
metrics.sh start/stop timers, value capture, RSS sampling,
percentile compute (for Phase D)
cases/
00_health.sh canary — gateway + 6 services /health → 200,
body identifies service, latency < 500ms (21 assertions)
fixtures/
csv/workers.csv spec's 5-row deterministic CSV
text/docs.txt 4 deterministic vector docs
expected/queries.json expected results for the 5 SQL assertions
Wired into the task runner:
just proof contract # canary only this commit
just proof integration # Phase C
just proof performance # Phase D
.gitignore: /tests/proof/reports/* with !.gitkeep — same pattern as
reports/scrum/_evidence/. Per-run output is a runtime artifact.
Specs landed alongside (J's drops):
docs/TEST_PROOF_SCOPE.md the harness contract this implements
docs/CLAUDE_REFACTOR_GUARDRAILS.md process discipline this harness obeys
Verified end-to-end (cached binaries):
just proof contract wall < 2s, 21 pass / 0 fail / 0 skip
just verify wall 31s, vet + test + 9 smokes still green
Two bugs fixed during canary run, both in run_proof.sh aggregation:
- grep -c exits 1 on zero matches; the `|| echo 0` form concatenated
"0\n0" and broke jq --argjson + integer comparison. Fixed via a
_count helper that captures count-or-zero cleanly.
- per-case table iterated case scripts (filename-based) but cases
write evidence under CASE_ID. Switched to JSONL-file iteration so
multi-case scripts work and the mapping is faithful.
Phase B (contract cases) lands next: 05_embedding, 06_vector_add,
08_gateway_contracts, 09_failure_modes. Each sourcing the same lib
helpers and writing to the same report shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,91 @@
|
||||
# tests/proof — claims-verification harness
|
||||
|
||||
Per `docs/TEST_PROOF_SCOPE.md`. The 9 smokes prove that the system *runs*; this harness proves that the system *makes the claims it claims to make*.
|
||||
|
||||
## Why this exists
|
||||
|
||||
Smokes verify that services boot, talk, and pass deterministic round-trips.
|
||||
They do not verify:
|
||||
|
||||
- contract drift (a route silently changes its response shape)
|
||||
- semantic correctness (the SQL query says what we claim it says)
|
||||
- failure-mode discipline (a malformed request returns 4xx, not silent 200)
|
||||
- performance regressions (vectors/sec drops 30% on a refactor)
|
||||
|
||||
The proof harness produces evidence, not pass/fail. Each case writes
|
||||
input/output hashes, latencies, status codes, log paths, git SHA → a
|
||||
future auditor can re-run + diff.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
tests/proof/
|
||||
README.md ← you are here
|
||||
claims.yaml ← enumeration of every claim, with id + type + routes
|
||||
run_proof.sh ← orchestrator (--mode contract|integration|performance)
|
||||
lib/
|
||||
env.sh ← service URLs, report dir, mode, git context
|
||||
http.sh ← curl wrappers (latency + status + body capture)
|
||||
assert.sh ← structured assertions writing JSONL evidence
|
||||
metrics.sh ← rss/cpu/timing capture for performance mode
|
||||
cases/
|
||||
00_health.sh
|
||||
01_storage_roundtrip.sh
|
||||
…
|
||||
10_perf_baseline.sh
|
||||
fixtures/
|
||||
csv/workers.csv ← canonical 5-row fixture (sha-pinned)
|
||||
text/docs.txt ← 4 deterministic vector docs
|
||||
expected/queries.json ← expected results for the 5 SQL assertions
|
||||
expected/rankings.json ← stored top-K rankings for vector search
|
||||
reports/
|
||||
proof-YYYYMMDD-HHMMSSZ/ ← per-run; gitignored
|
||||
summary.md
|
||||
summary.json
|
||||
raw/
|
||||
context.json ← git_sha, hostname, timestamp, mode
|
||||
cases/<id>.jsonl ← one JSONL line per assertion
|
||||
http/<id>/*.{json,body,headers}
|
||||
logs/<svc>.log ← captured stdout+stderr from booted services
|
||||
metrics/<id>.jsonl
|
||||
```
|
||||
|
||||
## Modes
|
||||
|
||||
```bash
|
||||
just proof contract # APIs, schemas, status codes; no big data; ~30s
|
||||
just proof integration # full chain CSV→storaged→…→queryd, text→embedd→vectord
|
||||
just proof performance # measurements only; runs after contract+integration
|
||||
```
|
||||
|
||||
The `just` recipes wrap `tests/proof/run_proof.sh` with `--mode <X>`. Use the script directly for advanced flags (`--no-bootstrap`, `--regenerate-rankings`, `--regenerate-baseline`).
|
||||
|
||||
## Hard rules (from TEST_PROOF_SCOPE.md)
|
||||
|
||||
- Don't claim performance without before/after metrics
|
||||
- Detect Ollama unavailability; mark embedding tests skipped or degraded with explanation
|
||||
- Skipped tests do not appear as passed
|
||||
- No silent ignore of missing services
|
||||
- No external cloud dependencies
|
||||
- No "HTTP 200" assertions unless the claim is health-only
|
||||
- No random data without a seed
|
||||
|
||||
## How to read a report
|
||||
|
||||
After `just proof integration`:
|
||||
|
||||
1. Open `tests/proof/reports/proof-<ts>/summary.md` for the human view.
|
||||
2. `summary.json` is the machine-readable counterpart.
|
||||
3. To investigate a single failed assertion:
|
||||
- find its `case_id` in `summary.md`
|
||||
- read `raw/cases/<case_id>.jsonl` (each line is one assertion)
|
||||
- cross-reference `raw/http/<case_id>/<probe>.{json,body,headers}` for the underlying HTTP round-trip
|
||||
|
||||
Every record cites the git SHA at run time; a clean re-run of the same SHA against the same fixtures must produce identical evidence (modulo timestamps + non-deterministic embedding noise).
|
||||
|
||||
## Reading order for new contributors
|
||||
|
||||
1. `docs/TEST_PROOF_SCOPE.md` — the spec this harness implements.
|
||||
2. `docs/CLAUDE_REFACTOR_GUARDRAILS.md` — process discipline this harness must obey when extended.
|
||||
3. `tests/proof/claims.yaml` — what's claimed.
|
||||
4. `tests/proof/cases/00_health.sh` — canonical case shape; copy-paste to add new cases.
|
||||
Reference in New Issue
Block a user