validatord: coordinator session JSONL for offline analysis (B follow-up)

Closes the second half of J's 2026-05-02 multi-call observability
concern. Trace-id propagation (commit d6d2fdf) gave us the *live*
view in Langfuse; this gives us the *longitudinal* view for ad-hoc
DuckDB queries over thousands of sessions:

  "show me every session where the model produced a real candidate
   without ever needing a retry"
  "find sessions where validation rejected three times in a row"
  "first-shot success rate per model — did we feed it enough corpus?"

## What's in

internal/validator/session_log.go:
  - SessionRecord type (schema=session.iterate.v1)
  - SessionLogger writer — mutex-guarded append, best-effort posture,
    nil-safe (NewSessionLogger("") = nil = no-op on Append)
  - BuildSessionRecord helper — assembles a row from any
    iterate response/failure/infra-error combination, callable from
    other daemons that wrap iterate (cross-daemon shared schema)
  - 7 unit tests including concurrent-append safety + the three
    code paths (success / max_iter_exhausted / infra_error)

cmd/validatord/main.go:
  - handlers.sessionLog field + wiring from cfg.Validatord.SessionLogPath
  - Iterate handler: build + append a SessionRecord on every call
  - rosterCheckFor("fill") closure stamps grounded_in_roster — the
    load-bearing forensic property J flagged ("we can never
    hallucinate available staff members to contracts")

internal/shared/config.go + lakehouse.toml:
  - [validatord].session_log_path field; empty = disabled
  - Production: /var/lib/lakehouse/validator/sessions.jsonl

scripts/validatord_smoke.sh:
  - Adds a probe verifying validatord announces session log path on
    startup. Smoke is now 6/6 (was 5/5).

docs/SESSION_LOG.md:
  - Schema reference + 5 worked DuckDB query examples including the
    "alarm" query (sessions where grounded_in_roster=false on an
    accepted fill — should always be empty; if not, something is
    bypassing FillValidator).

## What this is NOT

This is NOT a duplicate of replay_runs.jsonl. They're siblings:
  - replay_runs.jsonl: replay tool's per-task retrieval+model output
  - sessions.jsonl: validatord's per-iterate full retry chain +
    grounded-in-roster verdict

A single coordinator session can produce rows in both streams; the
session_id (= Langfuse trace_id) is the join key.

## Layered observability now in place

  Live view:  Langfuse trace tree (X-Lakehouse-Trace-Id propagation)
              `iterate.attempt[N]` spans with prompt/raw/verdict
  Offline:    coordinator_sessions.jsonl (this commit)
              DuckDB-queryable; longitudinal forensics
  Hard gate:  FillValidator + WorkerLookup (existing)
              phantom IDs structurally rejected, never reach
              session log's grounded_in_roster=true bucket

Per the architecture invariant in STATE_OF_PLAY's DO NOT RELITIGATE
section — these layers are wired; future work targets the data, not
the wiring.

## Verification

- internal/validator: 7 new tests (session_log_test.go) — all PASS
- cmd/validatord: 3 new integration tests covering the success,
  failure, and grounded=false paths — all PASS
- validatord_smoke.sh: 6/6 PASS through gateway :3110
- Full go test ./... green across 33 packages

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-05-02 05:22:09 -05:00
co-authored by Claude Opus 4.7
parent d6d2fdf81f
commit 1a3a82aedb
9 changed files with 848 additions and 7 deletions
+65 -1
View File
@@ -78,7 +78,11 @@ func main() {
// reference lets us emit child spans of the per-request trace.
// nil when Langfuse env isn't set; the iterate handler skips
// span emission gracefully in that case.
lf: shared.LoadLangfuseFromEnv(),
lf: shared.LoadLangfuseFromEnv(),
sessionLog: validator.NewSessionLogger(cfg.Validatord.SessionLogPath),
}
if h.sessionLog != nil {
slog.Info("validatord session log", "path", cfg.Validatord.SessionLogPath)
}
if err := shared.Run("validatord", cfg.Validatord.Bind, h.register, cfg.Auth); err != nil {
@@ -98,6 +102,10 @@ type handlers struct {
// creates the parent trace lives in shared.Run; this client lets
// validatord's handler emit application-level child spans.
lf *langfuse.Client
// sessionLog appends one SessionRecord per /v1/iterate session to
// `coordinator_sessions.jsonl` for offline DuckDB analysis. nil
// when the session log is unconfigured (best-effort posture).
sessionLog *validator.SessionLogger
}
func (h *handlers) register(r chi.Router) {
@@ -269,7 +277,16 @@ func (h *handlers) handleIterate(w http.ResponseWriter, r *http.Request) {
cfg.Tracer = h.iterTracer(r.Context())
}
t0 := time.Now()
resp, fail, err := validator.Iterate(r.Context(), req, cfg, chat, validate)
durationMs := time.Since(t0).Milliseconds()
// Emit one session row regardless of success/failure/infra-error.
// Best-effort: a failing session log never blocks the response.
rosterCheck := h.rosterCheckFor(req.Kind)
rec := validator.BuildSessionRecord(req, resp, fail, err, rosterCheck, durationMs)
h.sessionLog.Append(rec)
if err != nil {
http.Error(w, err.Error(), http.StatusBadGateway)
return
@@ -281,6 +298,53 @@ func (h *handlers) handleIterate(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, resp)
}
// rosterCheckFor returns a closure that verifies an accepted artifact's
// candidate IDs all exist in the roster. Used by BuildSessionRecord to
// stamp `grounded_in_roster` on each session row — answers the "did
// the model produce real workers, not phantoms?" question at offline-
// query time.
//
// Only the `fill` kind has worker IDs to verify. Other kinds return
// nil (omits the field from the session record).
//
// Note: FillValidator already enforces this constraint on every
// successful iteration (phantom IDs raise ValidationError::Consistency).
// Stamping the bool here surfaces the verified-grounded property as
// queryable data, not relying on validator semantics being recalled
// during analysis. A future scrum may want to query "show me sessions
// where the model guessed real worker IDs without ever needing a
// retry" — that's a `iterations=1 AND grounded_in_roster=true` query.
func (h *handlers) rosterCheckFor(kind string) func(map[string]any) *bool {
if kind != "fill" {
return nil
}
return func(artifact map[string]any) *bool {
fills, ok := artifact["fills"].([]any)
if !ok {
f := false
return &f
}
for _, raw := range fills {
fill, ok := raw.(map[string]any)
if !ok {
f := false
return &f
}
id, _ := fill["candidate_id"].(string)
if id == "" {
f := false
return &f
}
if _, found := h.lookup.Find(id); !found {
f := false
return &f
}
}
t := true
return &t
}
}
// chatCaller wires the iteration loop to chatd via HTTP. Builds the
// chat.Request shape, posts to ${chatdURL}/chat, returns the content
// string (no choices wrapper — chatd's response is already flat).