proof harness Phase C: 6 integration cases · 104/0/1 green

Adds the integration tier — full chain CSV→Parquet→SQL and full
text→embed→vector→search. All 10 cases (4 contract + 6 integration)
end-to-end deterministic; 8s wall total.

Cases added:
  01_storage_roundtrip.sh
    GOLAKE-010-012. PUT 1KiB → GET sha256-equal → LIST contains key
    → DELETE 200/204 → GET 404. Deterministic key under
    proof/<case_id>/ so concurrent runs don't collide.

  02_catalog_manifest.sh
    GOLAKE-020-022. Fresh register existing=false → manifest read
    matches → list contains dataset_id → idempotent re-register
    existing=true with stable dataset_id → schema-drift register
    409 (the ADR-020 contract). Per-run unique name via
    PROOF_RUN_ID so existing=false is meaningful.

  03_ingest_csv_to_parquet.sh
    GOLAKE-030. workers.csv (5 rows) via /v1/ingest multipart →
    parquet object on storaged → catalog manifest with row_count=5.
    Verifies content-addressed key shape (datasets/<n>/<fp>.parquet).

  04_query_correctness.sh
    GOLAKE-040. The 5 SQL assertions from fixtures/expected/queries.json
    against the workers fixture: count=5, Chicago=2, max=95,
    safety→Barbara, Houston avg=89.5. Iterates the YAML claims, runs
    each query, compares response columns to expected values.

  06_vector_add_search.sh integration extension
    GOLAKE-051. text → /v1/embed (4 docs from fixtures/text/docs.txt)
    → vectord add → search by query embedding. Top-1 ID per query
    asserted against fixtures/expected/rankings.json. First run (or
    --regenerate-rankings) writes the fixture and emits a skip with
    explicit reason; subsequent runs assert against it.

  07_vector_persistence_restart.sh
    GOLAKE-070. add 4 unit-basis vectors → search → record top-1
    distance → SIGTERM vectord → restart with the same --config →
    poll /health for 8s → search again → top-1 ID and distance match
    bit-identically. Skips with reason if vectord PID can't be found
    or post-restart bind times out.

Two harness improvements landed alongside:

  run_proof.sh writes a temp lakehouse_proof.toml with
  refresh_every="500ms" override and passes --config to all booted
  binaries. Production default is 30s; 04_query_correctness needs
  queryd to pick up the new view within a tick. Production config
  unchanged.

  cleanup() now pgreps for any orphan bin/<svc> processes (anchored
  to start-of-argv per memory feedback_pkill_scope.md) so a case
  that restarts a service mid-run still gets cleaned up.

  lib/http.sh adds proof_call(case_id, probe, method, url, args...)
  — escape hatch for cases that need raw curl args (multipart -F,
  custom headers). Used by 03_ingest for the multipart upload that
  conflicts with proof_post's --data + Content-Type defaults.

  lib/env.sh exports PROOF_RUN_ID — short unique id derived from the
  report directory timestamp. Used by 02 and 07 for fresh-each-run
  state isolation.

Two real findings recorded as evidence (no code changes):
  - rankings.json fixture pinned: 4 queries → 4 distinct top-1 docs
    via nomic-embed-text. A model swap that changes ranking now
    fails the harness loudly; --regenerate-rankings is the override.
  - vectord persistence kill+restart preserves top-1 distance
    bit-identically — the LHV1 single-Put framed format from
    G1P round-trips exactly through Save/Load.

Verified end-to-end:
  just proof contract       — 53 pass (4 cases)
  just proof integration    — 104 pass (10 cases) · 8s wall
  just verify               — 9 smokes still green · 33s wall

Phase D (performance baseline) lands next: 10_perf_baseline measures
rows/sec ingest, vectors/sec add, p50/p95 query+search latency, RSS,
CPU. First run writes tests/proof/baseline.json; later runs diff
against it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

tests/proof/run_proof.sh

@@ -74,9 +74,20 @@ PIDS=()
 WE_BOOTED=0
 
 cleanup() {
-    if [ "$WE_BOOTED" -eq 1 ] && [ "${#PIDS[@]}" -gt 0 ]; then
-        echo "[run_proof] cleanup: killing ${#PIDS[@]} services we started"
-        kill "${PIDS[@]}" 2>/dev/null || true
+    if [ "$WE_BOOTED" -eq 1 ]; then
+        # Kill the original PIDs we recorded plus any restarts a case
+        # might have done (07_vector_persistence_restart kills+restarts
+        # vectord mid-case, which orphans the original PID and creates
+        # a new one we never tracked). pgrep pattern is anchored to
+        # bin/<name> at start-of-argv per memory feedback_pkill_scope.md.
+        echo "[run_proof] cleanup: stopping services we started (incl. any restarts)"
+        if [ "${#PIDS[@]}" -gt 0 ]; then
+            kill "${PIDS[@]}" 2>/dev/null || true
+        fi
+        for svc in storaged catalogd ingestd queryd vectord embedd gateway; do
+            pgrep -f "^[./]*bin/${svc}($| )" 2>/dev/null \
+                | xargs -r kill 2>/dev/null || true
+        done
         wait 2>/dev/null || true
     fi
 }
@@ -101,6 +112,13 @@ bootstrap_services() {
         return 1
     fi
 
+    # Override queryd's refresh_every to 500ms so cases see new
+    # manifests within a tick — production default is 30s, which races
+    # against ingest→query cases. Default config left alone for prod.
+    local CFG_OVERRIDE="${PROOF_REPORT_DIR}/raw/lakehouse_proof.toml"
+    sed 's/^refresh_every *=.*/refresh_every = "500ms"/' lakehouse.toml > "$CFG_OVERRIDE"
+    export PROOF_LAKEHOUSE_CONFIG="$CFG_OVERRIDE"
+
     echo "[run_proof] bootstrap: launching services in dep order..."
     for SPEC in "storaged:3211" "catalogd:3212" "ingestd:3213" "queryd:3214" "vectord:3215" "embedd:3216" "gateway:3110"; do
         local NAME="${SPEC%:*}" PORT="${SPEC#*:}"
@@ -109,7 +127,8 @@ bootstrap_services() {
             echo "  ✓ ${NAME} (:${PORT}) already up — leaving as-is"
             continue
         fi
-        ./bin/"$NAME" > "${PROOF_REPORT_DIR}/raw/logs/${NAME}.log" 2>&1 &
+        ./bin/"$NAME" --config "$CFG_OVERRIDE" \
+            > "${PROOF_REPORT_DIR}/raw/logs/${NAME}.log" 2>&1 &
         PIDS+=("$!")
         if poll_health "$NAME" "$PORT"; then
             echo "  ✓ ${NAME} (:${PORT}) booted"