validatord: coordinator session JSONL for offline analysis (B follow-up)
Closes the second half of J's 2026-05-02 multi-call observability
concern. Trace-id propagation (commit d6d2fdf) gave us the *live*
view in Langfuse; this gives us the *longitudinal* view for ad-hoc
DuckDB queries over thousands of sessions:
"show me every session where the model produced a real candidate
without ever needing a retry"
"find sessions where validation rejected three times in a row"
"first-shot success rate per model — did we feed it enough corpus?"
## What's in
internal/validator/session_log.go:
- SessionRecord type (schema=session.iterate.v1)
- SessionLogger writer — mutex-guarded append, best-effort posture,
nil-safe (NewSessionLogger("") = nil = no-op on Append)
- BuildSessionRecord helper — assembles a row from any
iterate response/failure/infra-error combination, callable from
other daemons that wrap iterate (cross-daemon shared schema)
- 7 unit tests including concurrent-append safety + the three
code paths (success / max_iter_exhausted / infra_error)
cmd/validatord/main.go:
- handlers.sessionLog field + wiring from cfg.Validatord.SessionLogPath
- Iterate handler: build + append a SessionRecord on every call
- rosterCheckFor("fill") closure stamps grounded_in_roster — the
load-bearing forensic property J flagged ("we can never
hallucinate available staff members to contracts")
internal/shared/config.go + lakehouse.toml:
- [validatord].session_log_path field; empty = disabled
- Production: /var/lib/lakehouse/validator/sessions.jsonl
scripts/validatord_smoke.sh:
- Adds a probe verifying validatord announces session log path on
startup. Smoke is now 6/6 (was 5/5).
docs/SESSION_LOG.md:
- Schema reference + 5 worked DuckDB query examples including the
"alarm" query (sessions where grounded_in_roster=false on an
accepted fill — should always be empty; if not, something is
bypassing FillValidator).
## What this is NOT
This is NOT a duplicate of replay_runs.jsonl. They're siblings:
- replay_runs.jsonl: replay tool's per-task retrieval+model output
- sessions.jsonl: validatord's per-iterate full retry chain +
grounded-in-roster verdict
A single coordinator session can produce rows in both streams; the
session_id (= Langfuse trace_id) is the join key.
## Layered observability now in place
Live view: Langfuse trace tree (X-Lakehouse-Trace-Id propagation)
`iterate.attempt[N]` spans with prompt/raw/verdict
Offline: coordinator_sessions.jsonl (this commit)
DuckDB-queryable; longitudinal forensics
Hard gate: FillValidator + WorkerLookup (existing)
phantom IDs structurally rejected, never reach
session log's grounded_in_roster=true bucket
Per the architecture invariant in STATE_OF_PLAY's DO NOT RELITIGATE
section — these layers are wired; future work targets the data, not
the wiring.
## Verification
- internal/validator: 7 new tests (session_log_test.go) — all PASS
- cmd/validatord: 3 new integration tests covering the success,
failure, and grounded=false paths — all PASS
- validatord_smoke.sh: 6/6 PASS through gateway :3110
- Full go test ./... green across 33 packages
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
+65
-1
@@ -78,7 +78,11 @@ func main() {
|
||||
// reference lets us emit child spans of the per-request trace.
|
||||
// nil when Langfuse env isn't set; the iterate handler skips
|
||||
// span emission gracefully in that case.
|
||||
lf: shared.LoadLangfuseFromEnv(),
|
||||
lf: shared.LoadLangfuseFromEnv(),
|
||||
sessionLog: validator.NewSessionLogger(cfg.Validatord.SessionLogPath),
|
||||
}
|
||||
if h.sessionLog != nil {
|
||||
slog.Info("validatord session log", "path", cfg.Validatord.SessionLogPath)
|
||||
}
|
||||
|
||||
if err := shared.Run("validatord", cfg.Validatord.Bind, h.register, cfg.Auth); err != nil {
|
||||
@@ -98,6 +102,10 @@ type handlers struct {
|
||||
// creates the parent trace lives in shared.Run; this client lets
|
||||
// validatord's handler emit application-level child spans.
|
||||
lf *langfuse.Client
|
||||
// sessionLog appends one SessionRecord per /v1/iterate session to
|
||||
// `coordinator_sessions.jsonl` for offline DuckDB analysis. nil
|
||||
// when the session log is unconfigured (best-effort posture).
|
||||
sessionLog *validator.SessionLogger
|
||||
}
|
||||
|
||||
func (h *handlers) register(r chi.Router) {
|
||||
@@ -269,7 +277,16 @@ func (h *handlers) handleIterate(w http.ResponseWriter, r *http.Request) {
|
||||
cfg.Tracer = h.iterTracer(r.Context())
|
||||
}
|
||||
|
||||
t0 := time.Now()
|
||||
resp, fail, err := validator.Iterate(r.Context(), req, cfg, chat, validate)
|
||||
durationMs := time.Since(t0).Milliseconds()
|
||||
|
||||
// Emit one session row regardless of success/failure/infra-error.
|
||||
// Best-effort: a failing session log never blocks the response.
|
||||
rosterCheck := h.rosterCheckFor(req.Kind)
|
||||
rec := validator.BuildSessionRecord(req, resp, fail, err, rosterCheck, durationMs)
|
||||
h.sessionLog.Append(rec)
|
||||
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadGateway)
|
||||
return
|
||||
@@ -281,6 +298,53 @@ func (h *handlers) handleIterate(w http.ResponseWriter, r *http.Request) {
|
||||
writeJSON(w, http.StatusOK, resp)
|
||||
}
|
||||
|
||||
// rosterCheckFor returns a closure that verifies an accepted artifact's
|
||||
// candidate IDs all exist in the roster. Used by BuildSessionRecord to
|
||||
// stamp `grounded_in_roster` on each session row — answers the "did
|
||||
// the model produce real workers, not phantoms?" question at offline-
|
||||
// query time.
|
||||
//
|
||||
// Only the `fill` kind has worker IDs to verify. Other kinds return
|
||||
// nil (omits the field from the session record).
|
||||
//
|
||||
// Note: FillValidator already enforces this constraint on every
|
||||
// successful iteration (phantom IDs raise ValidationError::Consistency).
|
||||
// Stamping the bool here surfaces the verified-grounded property as
|
||||
// queryable data, not relying on validator semantics being recalled
|
||||
// during analysis. A future scrum may want to query "show me sessions
|
||||
// where the model guessed real worker IDs without ever needing a
|
||||
// retry" — that's a `iterations=1 AND grounded_in_roster=true` query.
|
||||
func (h *handlers) rosterCheckFor(kind string) func(map[string]any) *bool {
|
||||
if kind != "fill" {
|
||||
return nil
|
||||
}
|
||||
return func(artifact map[string]any) *bool {
|
||||
fills, ok := artifact["fills"].([]any)
|
||||
if !ok {
|
||||
f := false
|
||||
return &f
|
||||
}
|
||||
for _, raw := range fills {
|
||||
fill, ok := raw.(map[string]any)
|
||||
if !ok {
|
||||
f := false
|
||||
return &f
|
||||
}
|
||||
id, _ := fill["candidate_id"].(string)
|
||||
if id == "" {
|
||||
f := false
|
||||
return &f
|
||||
}
|
||||
if _, found := h.lookup.Find(id); !found {
|
||||
f := false
|
||||
return &f
|
||||
}
|
||||
}
|
||||
t := true
|
||||
return &t
|
||||
}
|
||||
}
|
||||
|
||||
// chatCaller wires the iteration loop to chatd via HTTP. Builds the
|
||||
// chat.Request shape, posts to ${chatdURL}/chat, returns the content
|
||||
// string (no choices wrapper — chatd's response is already flat).
|
||||
|
||||
Reference in New Issue
Block a user