gauntlet 2026-05-02: smoke chain + per-component scrum + parity probe
Production-readiness gauntlet exploiting the dual Rust/Go
implementation as a measurement instrument.
## Phase 1 — Full smoke chain
21/21 PASS in ~60s. Substrate intact across the full service surface.
## Phase 2 — Per-component scrum (token-volume fix)
Prior wave (165KB diff): Kimi 62 tokens out, Qwen 297 → no useful
analysis. This wave splits today's commits into 4 focused bundles
(36-71KB each):
c1 validatord (46KB) → 0 convergent / 11 distinct
c2 vectord substrate (36KB) → 0 convergent / 10 distinct
c3 materializer (71KB) → 0 convergent / 6 distinct (Opus emitted
a BLOCK then self-retracted in same response)
c4 replay (45KB) → 0 convergent / 10 distinct
Reviewer engagement vs prior wave: Kimi went 62 → ~250 tokens out
once bundles dropped below 60KB.
scripts/scrum_review.sh hardening:
* Diff-size guard (warn >60KB, hard-fail >100KB,
SCRUM_FORCE_OVERSIZE=1 override)
* Tightened prompt — file path must appear EXACTLY as in diff
so post-processor can grep WHERE: lines reliably
* Auto-tally step dedupes by (reviewer, location); convergence
counts distinct lineages (closes the prior `opus+opus+opus`
false-convergence bug)
## Phase 3 — Cross-runtime validator parity probe (the headline finding)
scripts/cutover/parity/validator_parity.sh sends 6 identical
/v1/validate cases to Rust :3100 AND Go :4110, compares status+body.
Result: **6/6 status codes match · 5/6 body shapes diverge.**
Rust returns serde-tagged enum: {"Schema":{"field":"x","reason":"y"}}
Go returns flat exported-fields: {"Kind":"schema","Field":"x","Reason":"y"}
Both round-trip inside their own runtime; a caller swapping one for
the other would break parsing silently. Captured as new _open_ row
in docs/ARCHITECTURE_COMPARISON.md decisions tracker.
This is the "use the dual-implementation as a measurement instrument"
return — single-repo scrums can't catch this class of cross-runtime
drift.
## Phase 4 — Production assessment
ship-with-known-gap. Validator wire-format gap is documented, not
regressed. ~50 LOC future fix on Go side (custom MarshalJSON on
ValidationError to match Rust's serde shape).
Persistent stack config (/tmp/lakehouse-persistent.toml) gains
validatord on :3221 + persistent-validatord binary so operators
bringing up the persistent stack get the new daemon automatically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
@@ -0,0 +1,22 @@
|
||||
# Scrum review — c1_validatord — kimi (openrouter/moonshotai/kimi-k2-0905)
|
||||
|
||||
**Latency:** 9132ms · **Tokens:** 14274 in / 179 out · **Date:** 2026-05-02
|
||||
|
||||
---
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: internal/validator/lookup_jsonl.go:86
|
||||
WHAT: silently skipping malformed JSON lines may hide operator errors
|
||||
WHY: missing malformed lines never alert ops; false positives stall validators downstream
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: internal/validator/playbook.go:72
|
||||
WHAT: ignores validation when `target_count` is missing instead of enforcing schema
|
||||
WHY: validation contract says “endorsed_names ≤ target_count×2”; absent value silences check, letting over-fills pass
|
||||
|
||||
SEVERITY: INFO
|
||||
WHERE: cmd/validatord/main.go:77
|
||||
WHAT: missing OpenAPI/JSONSchema tags attributed to external spec doc
|
||||
WHY: lack of machine-readable spec impedes client gen & evolution without human intent
|
||||
|
||||
VERDICT: ship-with-fixes — log malformed roster lines & require target_count in playbook artifact
|
||||
@@ -0,0 +1,37 @@
|
||||
# Scrum review — c1_validatord — opus (opencode/claude-opus-4-7)
|
||||
|
||||
**Latency:** 18077ms · **Tokens:** 6 in / 836 out · **Date:** 2026-05-02
|
||||
|
||||
---
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: cmd/validatord/main.go:178
|
||||
WHAT: mergeContext lets caller-supplied artifact._context override request context, opposite of comment in handleValidate flow.
|
||||
WHY: Comment says "caller-supplied artifact._context wins on key collision," but for /iterate the artifact is LLM-generated; trusting model output to override server context is a privilege-inversion risk for fields like client_id/target_count.
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: internal/validator/iterate.go:159
|
||||
WHAT: ExtractJSON's brace-balance scan ignores `{` and `}` inside JSON string literals.
|
||||
WHY: Inputs like `{"k":"a}b"}` break depth tracking — the `}` inside the string closes the object early and the remaining `"}` makes parseObject fail; cross-runtime parity claim with Rust depends on Rust having the same bug.
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: cmd/validatord/main.go:259
|
||||
WHAT: chatCaller reads entire response body with `io.ReadAll` with no size cap.
|
||||
WHY: A misbehaving/compromised chatd could return an unbounded body and OOM validatord; the request side caps at 4 MiB but the response side does not.
|
||||
|
||||
SEVERITY: INFO
|
||||
WHERE: internal/validator/iterate.go:121
|
||||
WHAT: `errSummary := vErr.Error()` feeds full error text back to the model unbounded.
|
||||
WHY: No truncation on the corrective prompt; pathological validators with long Reason strings could blow context windows, but unlikely in practice.
|
||||
|
||||
SEVERITY: INFO
|
||||
WHERE: internal/validator/iterate.go:235
|
||||
WHAT: `trim` duplicates the same helper in cmd/validatord/main.go.
|
||||
WHY: Two copies of a 4-line byte-prefix trimmer; harmless but worth consolidating into shared.
|
||||
|
||||
SEVERITY: INFO
|
||||
WHERE: cmd/validatord/main.go:117
|
||||
WHAT: Comment says validators "only ever return ValidationError" but coerces other errors to a Schema ValidationError with 422 instead of 500.
|
||||
WHY: Comment promises 500 ("Surface as 500 rather than silently coercing") but code returns the error as a ValidationError, which the caller renders as 422 — comment and behavior disagree.
|
||||
|
||||
VERDICT: ship-with-fixes — context-merge precedence + response size cap worth addressing.
|
||||
@@ -0,0 +1,17 @@
|
||||
# Scrum review — c1_validatord — qwen (openrouter/qwen/qwen3-coder)
|
||||
|
||||
**Latency:** 4028ms · **Tokens:** 14447 in / 143 out · **Date:** 2026-05-02
|
||||
|
||||
---
|
||||
|
||||
SEVERITY: BLOCK
|
||||
WHERE: cmd/validatord/main.go:210
|
||||
WHAT: chatCaller ignores provider parameter in HTTP request to chatd
|
||||
WHY: The chatCaller function hardcodes the URL path to "/chat" instead of using the provider parameter, potentially routing requests incorrectly.
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: internal/validator/iterate.go:157
|
||||
WHAT: Default temperature configuration may cause unintended behavior when set to zero
|
||||
WHY: When cfg.DefaultTemperature is exactly 0, the code assigns the defaultTemperature constant (0.2), which could mask intended zero values.
|
||||
|
||||
VERDICT: hold — critical routing bug in chatCaller, minor config fallback issue
|
||||
@@ -0,0 +1,27 @@
|
||||
# Convergence tally — c1_validatord
|
||||
|
||||
**Date:** 2026-05-02 · **Diff:** 1445 lines / 46329 bytes
|
||||
|
||||
## Findings by location
|
||||
|
||||
| Reviewers | Severity | Where | Hits |
|
||||
|---|---|---|---:|
|
||||
| kimi | INFO | `cmd/validatord/main.go:77` | 1 |
|
||||
| kimi | WARN | `internal/validator/lookup_jsonl.go:86` | 1 |
|
||||
| kimi | WARN | `internal/validator/playbook.go:72` | 1 |
|
||||
| opus | INFO | `cmd/validatord/main.go:117` | 1 |
|
||||
| opus | INFO | `internal/validator/iterate.go:121` | 1 |
|
||||
| opus | INFO | `internal/validator/iterate.go:235` | 1 |
|
||||
| opus | WARN | `cmd/validatord/main.go:178` | 1 |
|
||||
| opus | WARN | `cmd/validatord/main.go:259` | 1 |
|
||||
| opus | WARN | `internal/validator/iterate.go:159` | 1 |
|
||||
| qwen | BLOCK | `cmd/validatord/main.go:210` | 1 |
|
||||
| qwen | WARN | `internal/validator/iterate.go:157` | 1 |
|
||||
|
||||
(Convergent rows above are those whose Reviewers column contains a '+' — i.e. ≥2 lineages flagged the same location.)
|
||||
|
||||
## Verdict line per reviewer
|
||||
|
||||
- **opus**: VERDICT: ship-with-fixes — context-merge precedence + response size cap worth addressing.
|
||||
- **kimi**: VERDICT: ship-with-fixes — log malformed roster lines & require target_count in playbook artifact
|
||||
- **qwen**: VERDICT: hold — critical routing bug in chatCaller, minor config fallback issue
|
||||
@@ -0,0 +1,22 @@
|
||||
# Scrum review — c2_vectord_substrate — kimi (openrouter/moonshotai/kimi-k2-0905)
|
||||
|
||||
**Latency:** 9394ms · **Tokens:** 11091 in / 201 out · **Date:** 2026-05-02
|
||||
|
||||
---
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: internal/vectord/index.go:528
|
||||
WHAT: DecodeIndex makes a blind copy of index parameters without validating they make sense
|
||||
WHY: When loading a persisted index from unknown provenance we should at least sanity-check that Dimensions ≥1, M>0, etc. to prevent a corrupted envelope from silently propelling a runtime bug.
|
||||
|
||||
SEVERITY: INFO
|
||||
WHERE: cmd/vectord/main.go:85
|
||||
WHAT: unexported saveTask has confusing non-idiomatic API (trigger instead of Start/Sync etc.)
|
||||
WHY: “trigger” shadows verb-heavy context; Go code typically uses method names like start(), run(), safeRun().
|
||||
|
||||
SEVERITY: INFO
|
||||
WHERE: internal/vectord/index.go:52
|
||||
WHAT: comment references 2026-05-01 audit during run-time code
|
||||
WHY: non-actionable future date; reviewers can clean these inotify artefacts after bake-in.
|
||||
|
||||
VERDICT: ship
|
||||
@@ -0,0 +1,32 @@
|
||||
# Scrum review — c2_vectord_substrate — opus (opencode/claude-opus-4-7)
|
||||
|
||||
**Latency:** 21399ms · **Tokens:** 6 in / 927 out · **Date:** 2026-05-02
|
||||
|
||||
---
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: cmd/vectord/main.go:saveTask.trigger
|
||||
WHAT: Save errors leave inflight=true forever if pending was set, but actually re-loop; however an error with no pending requests correctly clears — yet a persistent failure plus ongoing triggers will spin in a tight loop logging on every iteration with no backoff.
|
||||
WHY: The `for` loop re-runs save immediately when pending is set; under sustained MinIO outage with sustained writes this becomes a hot retry loop saturating logs and a goroutine.
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: internal/vectord/index.go:rebuildGraphLocked
|
||||
WHAT: `g.Distance = i.g.Distance` reads from the (potentially corrupted) old graph; if i.g was nil-deref-corrupted, the Distance field may still be valid but this couples rebuild to old-graph state unnecessarily when params already encode the distance.
|
||||
WHY: The rebuild's stated load-bearing property is "independence from i.g state"; pulling Distance from i.g undermines that — should resolve via distanceFn(i.params.Distance) like NewIndex does.
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: internal/vectord/index.go:DecodeIndex
|
||||
WHAT: v1 fallback now silently drops ids whose meta exists but graph Lookup fails, where previously they were tracked in i.ids.
|
||||
WHY: Comment acknowledges the contract tightening, but there is no warning/log on dropped IDs — an operator decoding a v1 envelope with stale meta will see a silent shrink with no diagnostic.
|
||||
|
||||
SEVERITY: INFO
|
||||
WHERE: internal/vectord/index_test.go:TestAdd_RecoversFromPanickingGraph
|
||||
WHAT: Test name claims to prove rebuild from a panicking graph but never actually forces a panic — only exercises safeGraphAdd on a healthy graph then cleans up.
|
||||
WHY: Test body does not match docstring intent; coverage of the rebuild-after-panic path is asserted only indirectly via TestAdd_PastThreshold_SustainedReAdd.
|
||||
|
||||
SEVERITY: INFO
|
||||
WHERE: internal/vectord/index_test.go:TestAdd_SmallIndex_ConcurrentDistinctIDs
|
||||
WHAT: Missing function-doc comment delimiter — the comment block above the function is detached from a prior test's closing brace, making it look like a continuation comment.
|
||||
WHY: The diff shows `}` immediately followed by `// playbook_record pattern:` with no blank line and no func keyword between — readability/godoc hazard.
|
||||
|
||||
VERDICT: ship-with-fixes — saveTask retry-loop backoff and rebuild Distance independence worth addressing.
|
||||
@@ -0,0 +1,17 @@
|
||||
# Scrum review — c2_vectord_substrate — qwen (openrouter/qwen/qwen3-coder)
|
||||
|
||||
**Latency:** 5772ms · **Tokens:** 11419 in / 158 out · **Date:** 2026-05-02
|
||||
|
||||
---
|
||||
|
||||
SEVERITY: BLOCK
|
||||
WHERE: cmd/vectord/main.go:100
|
||||
WHAT: saveTask.trigger lacks error handling for returned error from save function
|
||||
WHY: The anonymous function inside saveTask.trigger calls save() but ignores its error return, only logging it, which violates the documented fail-open posture and could mask persistent save failures
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: internal/vectord/index.go:223
|
||||
WHAT: Re-add logic doesn't update metadata when replacing existing vector entries
|
||||
WHY: The code deletes from i.vectors but doesn't explicitly handle metadata updates for re-added IDs, potentially leaving stale metadata until the meta assignment at the end
|
||||
|
||||
VERDICT: ship-with-fixes | saveTask error handling, metadata consistency on re-add
|
||||
@@ -0,0 +1,26 @@
|
||||
# Convergence tally — c2_vectord_substrate
|
||||
|
||||
**Date:** 2026-05-02 · **Diff:** 966 lines / 35604 bytes
|
||||
|
||||
## Findings by location
|
||||
|
||||
| Reviewers | Severity | Where | Hits |
|
||||
|---|---|---|---:|
|
||||
| kimi | INFO | `cmd/vectord/main.go:85` | 1 |
|
||||
| kimi | INFO | `internal/vectord/index.go:52` | 1 |
|
||||
| kimi | WARN | `internal/vectord/index.go:528` | 1 |
|
||||
| opus | INFO | `internal/vectord/index_test.go:TestAdd_RecoversFromPanickingGraph` | 1 |
|
||||
| opus | INFO | `internal/vectord/index_test.go:TestAdd_SmallIndex_ConcurrentDistinctIDs` | 1 |
|
||||
| opus | WARN | `cmd/vectord/main.go:saveTask.trigger` | 1 |
|
||||
| opus | WARN | `internal/vectord/index.go:DecodeIndex` | 1 |
|
||||
| opus | WARN | `internal/vectord/index.go:rebuildGraphLocked` | 1 |
|
||||
| qwen | BLOCK | `cmd/vectord/main.go:100` | 1 |
|
||||
| qwen | WARN | `internal/vectord/index.go:223` | 1 |
|
||||
|
||||
(Convergent rows above are those whose Reviewers column contains a '+' — i.e. ≥2 lineages flagged the same location.)
|
||||
|
||||
## Verdict line per reviewer
|
||||
|
||||
- **opus**: VERDICT: ship-with-fixes — saveTask retry-loop backoff and rebuild Distance independence worth addressing.
|
||||
- **kimi**: VERDICT: ship
|
||||
- **qwen**: VERDICT: ship-with-fixes | saveTask error handling, metadata consistency on re-add
|
||||
@@ -0,0 +1,22 @@
|
||||
# Scrum review — c3_materializer — kimi (openrouter/moonshotai/kimi-k2-0905)
|
||||
|
||||
**Latency:** 15838ms · **Tokens:** 22537 in / 252 out · **Date:** 2026-05-02
|
||||
|
||||
---
|
||||
|
||||
SEVERITY: BLOCK
|
||||
WHERE: internal/materializer/validate.go:120
|
||||
WHAT: debug-only sha256 hash is computed but never stored or compared
|
||||
WHY: if caller relies on `human_override.decision` to gate non-trivial operations the string literal path used here (hash of entire record) will collide and create a false-positive match; deterministic failure path that bypasses operator review.
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: internal/materializer/canonical.go:56-59
|
||||
WHAT: map key ordering via insertion-order slice re-implements own JSON marshaler
|
||||
WHY: couples canonicalisation logic to a fragile home-grown JSON encoder; future refactors will drift from the stdlib/encoding/json implementation and break hash stability.
|
||||
|
||||
SEVERITY: INFO
|
||||
WHERE: cmd/materializer/main.go:78
|
||||
WHAT: uses os.Getenv and falls back to os.Getwd without checking getwd error – will silently drop to "." when inside an unreadable directory
|
||||
WHY: surface tool failure clearly rather than silently defaulting to cwd.
|
||||
|
||||
VERDICT: hold – BLOCK must be fixed before merge; sha256 collision bug breaks trusted override logic.
|
||||
@@ -0,0 +1,41 @@
|
||||
# Scrum review — c3_materializer — opus (opencode/claude-opus-4-7)
|
||||
|
||||
**Latency:** 21064ms · **Tokens:** 6 in / 1133 out · **Date:** 2026-05-02
|
||||
|
||||
---
|
||||
|
||||
SEVERITY: BLOCK
|
||||
WHERE: internal/materializer/materializer.go:processSource
|
||||
WHAT: loadSeenHashes is called BEFORE the output directory is created, but the seen-hash dedup must read outPath which lives in outDir; on first run this is fine (file doesn't exist), but skipsToWrite/rowsToWrite are accumulated in memory and flushed only at end — meaning a mid-run crash loses all the writes AND the next run re-processes them as new rather than dedup. Actually re-reading: writes are buffered then appended atomically; OK. Retracting. (No finding.)
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: internal/materializer/materializer.go:178
|
||||
WHAT: `strings.TrimPrefix(p, opts.Root+string(os.PathSeparator))` won't strip when Root has a trailing slash or differs in cleanliness, leaking absolute paths into receipt's output_files.
|
||||
WHY: filepath.Join may not preserve a user-supplied trailing separator; safer to use filepath.Rel(opts.Root, p).
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: internal/materializer/materializer.go:loadSeenHashes
|
||||
WHAT: Dedup only loads sig_hashes from today's partition file; identical rows reprocessed on a different RecordedAt date will be written again rather than deduped.
|
||||
WHY: isoDatePartition derives outDir from RecordedAt, so cross-day re-runs produce duplicate evidence rows for the same source line — breaks the idempotency contract claimed in the commit message.
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: internal/materializer/materializer.go:processSource
|
||||
WHAT: `lineOffset := int64(-1)` then `lineOffset++` before the empty-line `continue` means RowsRead matches non-empty lines but offsets in skip records skip empty lines too, while the TS port likely tracks raw line numbers.
|
||||
WHY: Operators correlating skip line_offset to source files will be off by the count of preceding blank lines.
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: internal/materializer/transforms.go:outcomesTransform
|
||||
WHAT: Test asserts `rec.ValidationResults["gap_signals"].(int64)` but the map is built with `numInt` returning int64 as `any`; works in-process, but JSON round-trip will produce float64 — fragile contract.
|
||||
WHY: ValidationResults is `map[string]any`; downstream consumers reading the marshaled JSON will see float64, mismatching the test's type assertion expectations.
|
||||
|
||||
SEVERITY: INFO
|
||||
WHERE: internal/materializer/transforms.go:buildRetrievedContext
|
||||
WHAT: Local variable shadows builtin `any` type alias.
|
||||
WHY: `any := false` shadows the universe-scope `any` (alias for interface{}); compiles but confuses readers and tooling.
|
||||
|
||||
SEVERITY: INFO
|
||||
WHERE: scripts/materializer_smoke.sh:34
|
||||
WHAT: `date -u +'%Y/%m/%d'` race: smoke test could straddle UTC midnight between materializer run and assertion.
|
||||
WHY: Flake risk; pin RecordedAt via env or compute once.
|
||||
|
||||
VERDICT: ship-with-fixes — cross-day dedup gap and absolute-path leak warrant follow-up.
|
||||
@@ -0,0 +1,17 @@
|
||||
# Scrum review — c3_materializer — qwen (openrouter/qwen/qwen3-coder)
|
||||
|
||||
**Latency:** 7736ms · **Tokens:** 23114 in / 176 out · **Date:** 2026-05-02
|
||||
|
||||
---
|
||||
|
||||
SEVERITY: BLOCK
|
||||
WHERE: internal/materializer/materializer.go:297
|
||||
WHAT: ValidationPass set false only when RowsSkipped > 0, but skips file write failures are ignored
|
||||
WHY: The receipt's ValidationPass is determined solely by `totals.RowsSkipped == 0`, but errors writing the skips file are silently dropped, which could lead to data loss without proper signaling.
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: internal/materializer/materializer.go:484
|
||||
WHAT: Fallback to "0000/00/00" partition may cause filesystem issues
|
||||
WHY: Using "0000/00/00" as a fallback for invalid timestamps can cause unexpected behavior, as it's not a valid date and could confuse downstream systems expecting real dates.
|
||||
|
||||
VERDICT: ship-with-fixes
|
||||
@@ -0,0 +1,14 @@
|
||||
# Convergence tally — c3_materializer
|
||||
|
||||
**Date:** 2026-05-02 · **Diff:** 2185 lines / 71382 bytes
|
||||
|
||||
## Findings by location
|
||||
|
||||
| Reviewers | Severity | Where | Hits |
|
||||
|---|---|---|---:|
|
||||
| opus | BLOCK | `internal/materializer/materializer.go:processSource` | 1 |
|
||||
| opus | INFO | `internal/materializer/transforms.go:buildRetrievedContext` | 1 |
|
||||
| opus | INFO | `scripts/materializer_smoke.sh:34` | 1 |
|
||||
| opus | WARN | `internal/materializer/materializer.go:178` | 1 |
|
||||
| opus | WARN | `internal/materializer/materializer.go:loadSeenHashes` | 1 |
|
||||
| opus | WARN | `internal/materializer/transforms.go:outcomesTransform` | 1 |
|
||||
@@ -0,0 +1,22 @@
|
||||
# Scrum review — c4_replay — kimi (openrouter/moonshotai/kimi-k2-0905)
|
||||
|
||||
**Latency:** 14239ms · **Tokens:** 13288 in / 203 out · **Date:** 2026-05-02
|
||||
|
||||
---
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: cmd/replay/main.go:87
|
||||
WHAT: Main exits 1 on validation failure AFTER printing full response, leaking potentially sensitive model outputs to stderr in automation.
|
||||
WHY: Fail-fast is good, but printing up to 1500+ chars of model response before exit invites secret/PII leakage into CI logs.
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: internal/replay/model.go:67
|
||||
WHAT: httpModelCaller hard-codes 180s timeout with no caller override, risking indefinite hang on slow gateway.
|
||||
WHY: Long fixed timeout hides infrastructure stalls and prevents graceful cancellation from higher-level contexts.
|
||||
|
||||
SEVERITY: INFO
|
||||
WHERE: internal/replay/retrieval.go:73
|
||||
WHAT: Malformed JSONL lines are silently skipped—no metric or log.
|
||||
WHY: Silent drop masks corpus corruption and complicates debugging.
|
||||
|
||||
VERDICT: ship-with-fixes - address stderr leak & timeout configurability.
|
||||
@@ -0,0 +1,32 @@
|
||||
# Scrum review — c4_replay — opus (opencode/claude-opus-4-7)
|
||||
|
||||
**Latency:** 18708ms · **Tokens:** 6 in / 926 out · **Date:** 2026-05-02
|
||||
|
||||
---
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: internal/replay/replay.go:67
|
||||
WHAT: RecordedRunID derived from RFC3339Nano timestamp can collide for back-to-back runs and is not unique per call.
|
||||
WHY: sha256(recordedAt)[:12] depends only on wall-clock time at nanosecond precision; two replays in the same nanosecond (or with monotonic clock quirks) produce identical run IDs, breaking the audit invariant.
|
||||
|
||||
SEVERITY: WARN
|
||||
WHERE: internal/replay/replay.go:90
|
||||
WHAT: Escalation reason prepend logic only fires when validation already passed, but is appended to whatever Reasons existed from the failing local call — message ordering implies the failure happened before recovery but the "recovered" note may sit alongside stale local-fail reasons.
|
||||
WHY: The code does `append([]string{"recovered..."}, validation.Reasons...)` after re-running ValidateResponse, which returns fresh Reasons (likely empty on Pass), so the note is fine — but if escalation Pass returns reasons (e.g., short response warning), the prefix conflates pass+warn states confusingly.
|
||||
|
||||
SEVERITY: INFO
|
||||
WHERE: internal/replay/retrieval.go:215
|
||||
WHAT: `trim` slices bytes of a UTF-8 string, can split a multi-byte rune and produce invalid UTF-8 in ContentPreview/ModelResponse log.
|
||||
WHY: `s[:n]` on a string with multi-byte characters at boundary n produces a malformed tail; replay corpora often contain non-ASCII (em dash, arrows used in this very repo's logs).
|
||||
|
||||
SEVERITY: INFO
|
||||
WHERE: internal/replay/replay.go:138
|
||||
WHAT: `persist.ReplayResult.ModelResponse = trim(...)` mutates a copy, but the comment says "log is truncated"; in-memory result returned to caller retains the full response — confirm this matches TS.
|
||||
WHY: Struct embedding copies by value here so the mutation is local to `persist`; behavior is correct but worth a one-line comment to prevent a future "fix".
|
||||
|
||||
SEVERITY: INFO
|
||||
WHERE: internal/replay/types.go:96
|
||||
WHAT: `DefaultLocalModel = "qwen3.5:latest"` and `qwen3.5:397b` in inferProvider — the `.5` form is unusual; verify the gateway accepts this exact tag.
|
||||
WHY: Smoke test asserts the literal string in stdout, so a typo would be caught at runtime, but the model name looks suspect compared to standard qwen2.5/qwen3 naming.
|
||||
|
||||
VERDICT: ship-with-fixes — run-id collision risk is the only real concern; rest is polish.
|
||||
@@ -0,0 +1,17 @@
|
||||
# Scrum review — c4_replay — qwen (openrouter/qwen/qwen3-coder)
|
||||
|
||||
**Latency:** 4870ms · **Tokens:** 13494 in / 110 out · **Date:** 2026-05-02
|
||||
|
||||
---
|
||||
|
||||
INFO
|
||||
WHERE: internal/replay/replay.go:45
|
||||
WHAT: time.Now() called twice in Replay function
|
||||
WHY: recordedAt uses time.Now().UTC().Format() while t0 uses time.Now() for duration calculation
|
||||
|
||||
INFO
|
||||
WHERE: internal/replay/replay.go:99
|
||||
WHAT: dryRunCaller appends [ESCALATED] suffix for parity with TypeScript
|
||||
WHY: comment indicates this is for test parity but may confuse users expecting actual escalation
|
||||
|
||||
VERDICT: ship | no issues found requiring blocking
|
||||
@@ -0,0 +1,26 @@
|
||||
# Convergence tally — c4_replay
|
||||
|
||||
**Date:** 2026-05-02 · **Diff:** 1308 lines / 44686 bytes
|
||||
|
||||
## Findings by location
|
||||
|
||||
| Reviewers | Severity | Where | Hits |
|
||||
|---|---|---|---:|
|
||||
| kimi | INFO | `internal/replay/retrieval.go:73` | 1 |
|
||||
| kimi | WARN | `cmd/replay/main.go:87` | 1 |
|
||||
| kimi | WARN | `internal/replay/model.go:67` | 1 |
|
||||
| opus | INFO | `internal/replay/replay.go:138` | 1 |
|
||||
| opus | INFO | `internal/replay/retrieval.go:215` | 1 |
|
||||
| opus | INFO | `internal/replay/types.go:96` | 1 |
|
||||
| opus | WARN | `internal/replay/replay.go:67` | 1 |
|
||||
| opus | WARN | `internal/replay/replay.go:90` | 1 |
|
||||
| qwen | | `internal/replay/replay.go:45` | 1 |
|
||||
| qwen | | `internal/replay/replay.go:99` | 1 |
|
||||
|
||||
(Convergent rows above are those whose Reviewers column contains a '+' — i.e. ≥2 lineages flagged the same location.)
|
||||
|
||||
## Verdict line per reviewer
|
||||
|
||||
- **opus**: VERDICT: ship-with-fixes — run-id collision risk is the only real concern; rest is polish.
|
||||
- **kimi**: VERDICT: ship-with-fixes - address stderr leak & timeout configurability.
|
||||
- **qwen**: VERDICT: ship | no issues found requiring blocking
|
||||
Reference in New Issue
Block a user