gauntlet 2026-05-02: smoke chain + per-component scrum + parity probe

Production-readiness gauntlet exploiting the dual Rust/Go
implementation as a measurement instrument.

## Phase 1 — Full smoke chain
21/21 PASS in ~60s. Substrate intact across the full service surface.

## Phase 2 — Per-component scrum (token-volume fix)
Prior wave (165KB diff): Kimi 62 tokens out, Qwen 297 → no useful
analysis. This wave splits today's commits into 4 focused bundles
(36-71KB each):
  c1 validatord (46KB) → 0 convergent / 11 distinct
  c2 vectord substrate (36KB) → 0 convergent / 10 distinct
  c3 materializer (71KB) → 0 convergent / 6 distinct (Opus emitted
                           a BLOCK then self-retracted in same response)
  c4 replay (45KB) → 0 convergent / 10 distinct

Reviewer engagement vs prior wave: Kimi went 62 → ~250 tokens out
once bundles dropped below 60KB.

scripts/scrum_review.sh hardening:
  * Diff-size guard (warn >60KB, hard-fail >100KB,
    SCRUM_FORCE_OVERSIZE=1 override)
  * Tightened prompt — file path must appear EXACTLY as in diff
    so post-processor can grep WHERE: lines reliably
  * Auto-tally step dedupes by (reviewer, location); convergence
    counts distinct lineages (closes the prior `opus+opus+opus`
    false-convergence bug)

## Phase 3 — Cross-runtime validator parity probe (the headline finding)
scripts/cutover/parity/validator_parity.sh sends 6 identical
/v1/validate cases to Rust :3100 AND Go :4110, compares status+body.

Result: **6/6 status codes match · 5/6 body shapes diverge.**

Rust returns serde-tagged enum:   {"Schema":{"field":"x","reason":"y"}}
Go returns flat exported-fields:  {"Kind":"schema","Field":"x","Reason":"y"}

Both round-trip inside their own runtime; a caller swapping one for
the other would break parsing silently. Captured as new _open_ row
in docs/ARCHITECTURE_COMPARISON.md decisions tracker.

This is the "use the dual-implementation as a measurement instrument"
return — single-repo scrums can't catch this class of cross-runtime
drift.

## Phase 4 — Production assessment
ship-with-known-gap. Validator wire-format gap is documented, not
regressed. ~50 LOC future fix on Go side (custom MarshalJSON on
ValidationError to match Rust's serde shape).

Persistent stack config (/tmp/lakehouse-persistent.toml) gains
validatord on :3221 + persistent-validatord binary so operators
bringing up the persistent stack get the new daemon automatically.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-05-02 04:05:18 -05:00
co-authored by Claude Opus 4.7
parent f9e72412c1
commit e8cf113af8
28 changed files with 7195 additions and 5 deletions
@@ -0,0 +1,332 @@
[d1-smoke] building...
[d1-smoke] launching in dep order...
[d1-smoke] /health probes:
✓ gateway (:3110) → {"status":"ok","service":"gateway"}
✓ storaged (:3211) → {"status":"ok","service":"storaged"}
✓ catalogd (:3212) → {"status":"ok","service":"catalogd"}
✓ ingestd (:3213) → {"status":"ok","service":"ingestd"}
✓ queryd (:3214) → {"status":"ok","service":"queryd"}
[d1-smoke] gateway proxy probes (D6+):
✓ POST /v1/ingest (no name) → 400 from ingestd (proxy wired)
✓ POST /v1/sql (no body) → 400 from queryd (proxy wired)
[d1-smoke] D1 acceptance gate: PASSED
[d1-smoke] cleanup
[d2-smoke] building storaged...
[d2-smoke] launching storaged...
[d2-smoke] PUT round-trip:
✓ PUT d2-smoke/1777712027.bin → 200
[d2-smoke] GET echoes bytes:
✓ GET d2-smoke/1777712027.bin → bytes match
[d2-smoke] LIST includes key:
✓ LIST prefix=d2-smoke/ → contains d2-smoke/1777712027.bin
[d2-smoke] DELETE then GET → 404:
✓ DELETE then GET → 404
[d2-smoke] 256 MiB cap → 413:
✓ PUT 257 MiB → 413
[d2-smoke] semaphore: 5th concurrent PUT → 503 + Retry-After:5
✓ 5th concurrent PUT → 503 + Retry-After: 5
[d2-smoke] D2 acceptance gate: PASSED
[d2-smoke] cleanup
[d3-smoke] building storaged + catalogd...
[d3-smoke] launching storaged...
[d3-smoke] launching catalogd (first start, empty catalog)...
[d3-smoke] POST /catalog/register (fresh):
✓ fresh register → existing=false, dataset_id=200a05a8-4f66-5a86-bdac-e17d87176613
[d3-smoke] GET /catalog/manifest/d3_smoke_dataset:
✓ manifest dataset_id matches
[d3-smoke] GET /catalog/list (1 entry):
✓ list count=1
[d3-smoke] restart catalogd → rehydrate from Parquet:
✓ rehydrated dataset_id matches across restart
[d3-smoke] re-register (same name + same fingerprint) → existing=true:
✓ existing=true, same dataset_id, objects replaced (count=2)
[d3-smoke] re-register (different fingerprint) → 409:
✓ different fingerprint → 409 Conflict
[d3-smoke] D3 acceptance gate: PASSED
[d3-smoke] cleanup
[d4-smoke] building storaged + catalogd + ingestd...
[d4-smoke] launching storaged → catalogd → ingestd...
[d4-smoke] POST /ingest?name=d4_workers (5 rows, 5 cols):
✓ ingest fresh → row_count=5, existing=false, key=datasets/d4_workers/247165ad7d53e8d5993d3181dc9ce9b1d06383b336c31c999a89bd48d41308a4.parquet
[d4-smoke] mc shows the parquet on MinIO:
✓ 247165ad7d53e8d5993d3181dc9ce9b1d06383b336c31c999a89bd48d41308a4.parquet present in lakehouse-go-primary/datasets/d4_workers/
[d4-smoke] catalogd manifest matches:
✓ manifest row_count=5, fp matches, 1 object at datasets/d4_workers/247165ad7d53e8d5993d3181dc9ce9b1d06383b336c31c999a89bd48d41308a4.parquet
[d4-smoke] ADR-010 — salary is string (mixed N/A):
✓ deferred to fingerprint stability (next test)
[d4-smoke] re-ingest same CSV → existing=true:
✓ idempotent re-ingest: existing=true, same dataset_id, same fingerprint
[d4-smoke] schema-drift CSV → 409:
✓ schema drift → 409 Conflict
[d4-smoke] D4 acceptance gate: PASSED
[d4-smoke] cleanup
[d5-smoke] building all 4 backing services...
[d5-smoke] launching storaged → catalogd → ingestd...
[d5-smoke] ingest 5-row CSV via D4 path:
✓ ingest row_count=5
[d5-smoke] launching queryd (initial Refresh picks up d5_workers)...
[d5-smoke] POST /sql SELECT count(*) FROM d5_workers:
✓ count(*)=5
[d5-smoke] POST /sql SELECT * FROM d5_workers LIMIT 3:
✓ rows[0] = (id=1, name=Alice), columns=[id, name, salary]
[d5-smoke] schema-drift ingest 409s; existing view still queries:
✓ drift → 409
✓ post-drift count(*)=5 (view unchanged)
[d5-smoke] error path: SELECT FROM nonexistent → 400:
✓ unknown table → 400
[d5-smoke] D5 acceptance gate: PASSED
[d5-smoke] cleanup
[d6-smoke] building all 5 binaries...
[d6-smoke] launching storaged → catalogd → ingestd...
[d6-smoke] launching gateway:
[d6-smoke] /v1/ingest?name=d6_workers (gateway → ingestd):
✓ ingest row_count=3, content-addressed key
[d6-smoke] /v1/catalog/list (gateway → catalogd):
✓ catalog count=1
[d6-smoke] /v1/storage/list?prefix=datasets/d6_workers/ (gateway → storaged):
✓ storage list returned 1 object(s) under datasets/d6_workers/
[d6-smoke] /v1/sql SELECT count(*) (gateway → queryd):
✓ count(*)=3
[d6-smoke] /v1/sql with row data (full round-trip):
✓ rows[0].name=Alice (full ingest → storage → catalog → query through gateway)
[d6-smoke] /v1/unknown → 404:
✓ unknown route → 404
[d6-smoke] D6 acceptance gate: PASSED
[d6-smoke] cleanup
[g1-smoke] building vectord + gateway...
[g1-smoke] launching vectord → gateway...
[g1-smoke] /v1/vectors/index — create dim=8:
✓ create → 201
[g1-smoke] duplicate create → 409:
✓ duplicate → 409
[g1-smoke] add batch of 200 vectors:
✓ added=200, length=200
[g1-smoke] search for inserted vector w-042 → recall:
✓ top hit = w-042 (dist=5.9604645E-8), 3 results, metadata round-tripped
[g1-smoke] dim mismatch on add → 400:
✓ dim mismatch → 400
[g1-smoke] search on missing index → 404:
✓ unknown index → 404
[g1-smoke] DELETE then GET → 404:
✓ post-delete GET → 404
[g1-smoke] G1 acceptance gate: PASSED
[g1-smoke] cleanup
[g1p-smoke] building storaged + vectord + gateway...
[g1p-smoke] launching storaged...
[g1p-smoke] launching vectord (round 1) → gateway...
[g1p-smoke] create index + add 50 vectors:
✓ added 50 → length=50
[g1p-smoke] verify storaged has the persistence file:
✓ _vectors/persist_demo.lhv1 present in storaged
[g1p-smoke] search pre-restart:
✓ pre-restart top hit = w-001
[g1p-smoke] kill + restart vectord (rehydrate path):
[g1p-smoke] vectord rehydrated index list shows persist_demo:
✓ list count=1 after restart
✓ length=50 after restart (state survived)
[g1p-smoke] search post-restart:
✓ post-restart top hit = w-001 (dist=0)
[g1p-smoke] DELETE then restart → index gone:
✓ persistence file removed from storaged
✓ post-delete restart list count=0
[g1p-smoke] G1P acceptance gate: PASSED
[g1p-smoke] cleanup
[g2-smoke] building embedd + vectord + gateway...
[g2-smoke] launching embedd → vectord (no persist) → gateway...
[g2-smoke] /v1/embed — two distinct texts:
✓ dim=768, model=nomic-embed-text-v2-moe, 2 distinct vectors
[g2-smoke] determinism — same text twice → byte-identical vector:
✓ identical text → identical vector
[g2-smoke] empty texts → 400:
✓ empty → 400
[g2-smoke] bad model → 502:
✓ unknown model → 502
[g2-smoke] end-to-end: embed → vectord add → search by embed → recall:
✓ embed → store → search round-trip: w-0 at dist=0
[g2-smoke] G2 acceptance gate: PASSED
[g2-smoke] cleanup
[chatd-smoke] building chatd + gateway...
[chatd-smoke] launching chatd → gateway...
[chatd-smoke] /v1/chat/providers — only ollama registered:
✓ exactly 1 provider (ollama, available=true)
[chatd-smoke] POST /v1/chat with bare model name:
✓ provider=ollama, latency=11134ms, content=ok…
[chatd-smoke] POST /v1/chat with explicit ollama/ prefix:
✓ ollama/qwen3.5:latest → provider=ollama, model=qwen3.5:latest (prefix stripped)
[chatd-smoke] POST /v1/chat with :cloud suffix (no cloud provider):
✓ kimi-k2.6:cloud → 404 (ollama_cloud not registered, no silent fall-through to local)
[chatd-smoke] POST /v1/chat with unknown/ prefix (falls through, upstream 502s):
✓ unknown/<model> → ollama default → upstream 502 (no silent prefix-strip)
[chatd-smoke] POST /v1/chat with missing model field:
✓ missing model → 400
[chatd-smoke] chatd acceptance gate: PASSED (6/6)
[chatd-smoke] cleanup
[downgrade-smoke] building matrixd + vectord + gateway...
[downgrade-smoke] launching vectord → matrixd → gateway...
[downgrade-smoke] strong model + no force → downgrade fires:
✓ codereview_lakehouse → codereview_isolation (downgraded_from=lakehouse)
[downgrade-smoke] forced_mode=true bypasses:
✓ caller-forced mode preserved, no downgrade
[downgrade-smoke] force_full_override=true bypasses:
✓ env-override bypass, no downgrade
[downgrade-smoke] weak model (qwen3.5:latest) bypasses:
✓ weak model keeps lakehouse
[downgrade-smoke] non-lakehouse mode → gate not applicable:
✓ codereview_isolation passes through unchanged
[downgrade-smoke] empty mode → 400:
✓ empty mode → 400
[downgrade-smoke] Downgrade gate acceptance: PASSED
[downgrade-smoke] cleanup
[matrix-smoke] building matrixd + vectord + gateway...
[matrix-smoke] launching vectord → matrixd → gateway...
[matrix-smoke] create two corpora:
✓ corpus_a and corpus_b created
[matrix-smoke] add vectors to both corpora:
✓ 3 + 3 vectors loaded
[matrix-smoke] /matrix/corpora lists both:
✓ count=2, both corpora listed
[matrix-smoke] /matrix/search multi-corpus retrieve+merge:
✓ 4 merged results · 3+3 per-corpus · both corpora represented
[matrix-smoke] top hit comes from corpus_b (b-near is globally closest):
✓ top hit: id=b-near corpus=corpus_b (closer than corpus_a's a-near)
[matrix-smoke] metadata preserved on merged results:
✓ metadata.label round-trips through matrix
[matrix-smoke] results sorted by distance ascending:
✓ distances ascending
[matrix-smoke] empty corpora → 400:
[matrix-smoke] missing corpus name → 502:
[matrix-smoke] no query (empty text and vector) → 400:
✓ empty=400, missing-corpus=502, no-query=400
[matrix-smoke] metadata_filter drops non-matching results:
✓ filter kept 2 ('a near' + 'b near'), dropped 4 mid/far entries
[matrix-smoke] Matrix acceptance gate: PASSED
[matrix-smoke] cleanup
[observer-smoke] building observerd + gateway...
[observer-smoke] launching observerd → gateway...
[observer-smoke] record 5 ops:
✓ 5 events posted
[observer-smoke] /observer/stats aggregates correctly:
✓ total=5 (3 ok + 2 fail) · by_source: mcp=3 scenario=2 · 2 scenario digests
[observer-smoke] empty endpoint → 400:
✓ empty endpoint rejected
[observer-smoke] kill + restart observerd → ops survive:
✓ total=5 ok=3 err=2 preserved through restart
[observer-smoke] Observer acceptance gate: PASSED
[observer-smoke] cleanup
[pathway-smoke] building pathwayd + gateway...
[pathway-smoke] launching pathwayd → gateway...
[pathway-smoke] Add → fresh UID + replay_count=1:
✓ uid=27f05e1f-4fee-4e8d-9409-9b7493ef9200 replay_count=1
[pathway-smoke] Get → returns same trace:
✓ content.approach round-trips
[pathway-smoke] AddIdempotent same UID → replay_count++:
✓ replay_count bumped to 2
[pathway-smoke] Update → in-place content replace:
✓ Update applied and persisted
[pathway-smoke] Revise → new UID with predecessor link:
✓ revision uid=9826a9d0-55f9-4fa7-b342-1bf692966d1a predecessor=27f05e1f-4fee-4e8d-9409-9b7493ef9200
[pathway-smoke] History → walks chain backward:
✓ chain length=2, [0]=9826a9d0-55f9-4fa7-b342-1bf692966d1a [1]=27f05e1f-4fee-4e8d-9409-9b7493ef9200
[pathway-smoke] Search tag=staffing → finds both traces:
✓ tag search count=2
[pathway-smoke] Retire → excluded from Search but Get-able:
✓ retired excluded from default Search, included with flag, still Get-able
[pathway-smoke] Stats → total/active/retired counters:
✓ total=2 active=1 retired=1
[pathway-smoke] Negative paths → 4xx semantics:
✓ get/update/revise/retire on unknown → 404; bad content → 400
[pathway-smoke] kill + restart pathwayd → state survives:
✓ replay_count, retired flag, predecessor link all preserved
[pathway-smoke] Pathway acceptance gate: PASSED
[pathway-smoke] cleanup
[playbook-smoke] building stack...
[playbook-smoke] launching embedd → vectord → matrixd → gateway...
[playbook-smoke] embedding 3 corpus items + query...
[playbook-smoke] create corpus widgets + add 3 items...
[playbook-smoke] baseline search (no playbook):
baseline order: widget-a,widget-b,widget-c widget-c distance=0.6565746
[playbook-smoke] record playbook: (alpha staffing query test full prompt) → widget-c score=1.0
✓ playbook_id=pb-4f1d0dccdb1df0ae
[playbook-smoke] boosted search (use_playbook=true):
boosted order: widget-a,widget-c,widget-b widget-c distance=0.3282873 playbook_boosted=1
✓ playbook_boosted=1 ≥ 1
widget-c distance ratio (boosted/baseline) = 0.5 (expect ≈ 0.5)
✓ ratio in [0.40, 0.60] — boost applied correctly
[playbook-smoke] bulk record 3 entries:
✓ 2 recorded, 1 failed (empty query_text caught), per-entry IDs/errors returned
[playbook-smoke] Playbook acceptance gate: PASSED
[playbook-smoke] cleanup
[relevance-smoke] building matrixd + vectord + gateway...
[relevance-smoke] launching vectord → matrixd → gateway...
[relevance-smoke] adjacency-pollution: Connector outranks Registry, junk dropped:
✓ Connector kept, junk dropped, Connector (0.6799999999999999) > Registry (-0.45555555555555555)
[relevance-smoke] empty chunks → 400:
✓ 400 on empty chunks
[relevance-smoke] threshold=10 (impossibly high) drops everything:
✓ threshold=10 drops everything (0 kept / 1 dropped)
[relevance-smoke] Relevance acceptance gate: PASSED
[relevance-smoke] cleanup
[cap-smoke] building storaged + gateway...
[cap-smoke] launching storaged → gateway...
[cap-smoke] generating 300 MiB deterministic payload...
size=314572800 sha=17a88af83717...
[cap-smoke] Test 1: PUT 300 MiB to _vectors/ (should pass)
✓ PUT _vectors/ → 200
[cap-smoke] Test 2: PUT 300 MiB to datasets/ (should reject)
✓ PUT datasets/ → 413 (default cap protects routine prefixes)
[cap-smoke] Test 3: GET _vectors/ — sha matches input
✓ GET round-trip preserves bytes (size=314572800 sha=17a88af83717)
[cap-smoke] ✓ Storaged cap smoke: PASSED
[cap-smoke] cleanup
[workflow-smoke] building observerd + gateway...
[workflow-smoke] launching observerd → gateway...
[workflow-smoke] /observer/workflow/modes lists fixtures + real modes:
✓ all 7 expected modes registered (fixtures + 4 pure + matrix.search HTTP)
[workflow-smoke] 3-node DAG: shape (upper) → weakness → improvement
✓ status=succeeded · shape=HELLO WORLD · refs propagated through 3-node chain
[workflow-smoke] /observer/stats reflects workflow ops:
✓ 3 workflow ops recorded (one per node), total=3
[workflow-smoke] unknown mode → 400:
✓ unknown mode aborts with 400 + helpful error
[workflow-smoke] real-mode chain: downgrade → distillation.score
✓ downgrade flipped lakehouse→isolation; scorer rated scrum_review attempt_1=accepted
[workflow-smoke] Workflow runner acceptance: PASSED
[workflow-smoke] cleanup
[materializer-smoke] building bin/materializer...
[materializer-smoke] dry-run probe
[materializer-smoke] first run
[evidence_index] 4 read · 3 written · 1 skipped · 0 deduped
data/_kb/distilled_facts.jsonl: read=3 wrote=2 skip=1 dedup=0
data/_kb/distilled_procedures.jsonl: (missing — skipped)
data/_kb/distilled_config_hints.jsonl: (missing — skipped)
data/_kb/contract_analyses.jsonl: (missing — skipped)
data/_kb/mode_experiments.jsonl: (missing — skipped)
data/_kb/scrum_reviews.jsonl: (missing — skipped)
data/_kb/observer_escalations.jsonl: read=1 wrote=1 skip=0 dedup=0
data/_kb/audit_facts.jsonl: (missing — skipped)
data/_kb/auto_apply.jsonl: (missing — skipped)
data/_kb/observer_reviews.jsonl: (missing — skipped)
data/_kb/audits.jsonl: (missing — skipped)
data/_kb/outcomes.jsonl: (missing — skipped)
[evidence_index] receipt: /tmp/tmp.eOKwqXIezb/reports/distillation/2026-05-02T08-54-40-881776326Z/receipt.json
[evidence_index] validation_pass=false
[materializer-smoke] idempotent re-run
[materializer-smoke] PASS
[replay-smoke] building bin/replay...
[replay-smoke] dry-run (with retrieval)
[replay-smoke] dry-run (no retrieval)
[replay-smoke] forced-fail with escalation
[replay-smoke] PASS
[validatord-smoke] building validatord + gateway...
[validatord-smoke] launching validatord → gateway...
✓ validatord roster loaded with 3 records
[validatord-smoke] /v1/validate playbook happy path:
✓ playbook OK ({"findings":[],"elapsed_ms":0})
[validatord-smoke] /v1/validate playbook missing fingerprint → 422:
✓ playbook missing fingerprint → 422 schema/fingerprint
[validatord-smoke] /v1/validate fill with phantom candidate → 422:
✓ phantom candidate W-PHANTOM → 422 consistency
[validatord-smoke] /v1/validate unknown kind → 400:
✓ unknown kind → 400
[validatord-smoke] PASS — 5/5 probes through gateway :3110
[validatord-smoke] cleanup
@@ -0,0 +1,22 @@
PASS d1 5s
PASS d2 21s
PASS d3 1s
PASS d4 1s
PASS d5 1s
PASS d6 1s
PASS g1 0s
PASS g1p 2s
PASS g2 5s
PASS chatd 12s
PASS downgrade 1s
PASS matrix 0s
PASS observer 1s
PASS pathway 2s
PASS playbook 1s
PASS relevance 1s
PASS storaged_cap 3s
PASS workflow 0s
PASS materializer 0s
PASS replay 1s
PASS validatord 0s
--- 21 PASS / 0 FAIL ---