validatord: coordinator session JSONL for offline analysis (B follow-up)

Closes the second half of J's 2026-05-02 multi-call observability
concern. Trace-id propagation (commit d6d2fdf) gave us the *live*
view in Langfuse; this gives us the *longitudinal* view for ad-hoc
DuckDB queries over thousands of sessions:

  "show me every session where the model produced a real candidate
   without ever needing a retry"
  "find sessions where validation rejected three times in a row"
  "first-shot success rate per model — did we feed it enough corpus?"

## What's in

internal/validator/session_log.go:
  - SessionRecord type (schema=session.iterate.v1)
  - SessionLogger writer — mutex-guarded append, best-effort posture,
    nil-safe (NewSessionLogger("") = nil = no-op on Append)
  - BuildSessionRecord helper — assembles a row from any
    iterate response/failure/infra-error combination, callable from
    other daemons that wrap iterate (cross-daemon shared schema)
  - 7 unit tests including concurrent-append safety + the three
    code paths (success / max_iter_exhausted / infra_error)

cmd/validatord/main.go:
  - handlers.sessionLog field + wiring from cfg.Validatord.SessionLogPath
  - Iterate handler: build + append a SessionRecord on every call
  - rosterCheckFor("fill") closure stamps grounded_in_roster — the
    load-bearing forensic property J flagged ("we can never
    hallucinate available staff members to contracts")

internal/shared/config.go + lakehouse.toml:
  - [validatord].session_log_path field; empty = disabled
  - Production: /var/lib/lakehouse/validator/sessions.jsonl

scripts/validatord_smoke.sh:
  - Adds a probe verifying validatord announces session log path on
    startup. Smoke is now 6/6 (was 5/5).

docs/SESSION_LOG.md:
  - Schema reference + 5 worked DuckDB query examples including the
    "alarm" query (sessions where grounded_in_roster=false on an
    accepted fill — should always be empty; if not, something is
    bypassing FillValidator).

## What this is NOT

This is NOT a duplicate of replay_runs.jsonl. They're siblings:
  - replay_runs.jsonl: replay tool's per-task retrieval+model output
  - sessions.jsonl: validatord's per-iterate full retry chain +
    grounded-in-roster verdict

A single coordinator session can produce rows in both streams; the
session_id (= Langfuse trace_id) is the join key.

## Layered observability now in place

  Live view:  Langfuse trace tree (X-Lakehouse-Trace-Id propagation)
              `iterate.attempt[N]` spans with prompt/raw/verdict
  Offline:    coordinator_sessions.jsonl (this commit)
              DuckDB-queryable; longitudinal forensics
  Hard gate:  FillValidator + WorkerLookup (existing)
              phantom IDs structurally rejected, never reach
              session log's grounded_in_roster=true bucket

Per the architecture invariant in STATE_OF_PLAY's DO NOT RELITIGATE
section — these layers are wired; future work targets the data, not
the wiring.

## Verification

- internal/validator: 7 new tests (session_log_test.go) — all PASS
- cmd/validatord: 3 new integration tests covering the success,
  failure, and grounded=false paths — all PASS
- validatord_smoke.sh: 6/6 PASS through gateway :3110
- Full go test ./... green across 33 packages

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-05-02 05:22:09 -05:00
co-authored by Claude Opus 4.7
parent d6d2fdf81f
commit 1a3a82aedb
9 changed files with 848 additions and 7 deletions
+16 -1
View File
@@ -70,6 +70,7 @@ roster_path = "$ROSTER"
default_max_iterations = 3
default_max_tokens = 4096
chat_timeout_secs = 240
session_log_path = "$TMP/sessions.jsonl"
EOF
poll_health() {
@@ -153,4 +154,18 @@ if [ "$STATUS" != "400" ]; then
fi
echo " ✓ unknown kind → 400"
echo "[validatord-smoke] PASS — 5/5 probes through gateway :3110"
# 6. Session log: every /v1/validate hit was a single-shot validation
# (not iterate), so the session log isn't populated yet. Verify it
# exists as a path (logger initialized) — the iterate handler is
# where rows actually land. This proves the wiring is in place
# even if the smoke doesn't drive a live iteration end-to-end.
LOG_PATH="$TMP/sessions.jsonl"
echo "[validatord-smoke] session log path wired:"
grep -q "validatord session log" /tmp/validatord.log || {
echo " ✗ expected validatord to log 'validatord session log' on startup"
grep validatord /tmp/validatord.log
exit 1
}
echo " ✓ session_log_path=$LOG_PATH announced at startup (rows land on /v1/iterate calls)"
echo "[validatord-smoke] PASS — 6/6 probes through gateway :3110"