profit/golangLAKEHOUSE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

trace-id propagation through /v1/iterate (multi-call observability)

Browse Source 
Closes J's 2026-05-02 multi-call observability gap: a single
/v1/iterate session with N retries used to surface in Langfuse as
N+1 disconnected traces (one per /v1/chat hop + one for the iterate
request itself), with no parent/child linkage. Operators couldn't
scroll the retry chain in one trace tree to spot where grounding
failed.

## Wire-level change

- New header constant `shared.TraceIDHeader = "X-Lakehouse-Trace-Id"`
- `langfuseMiddleware` honors the header on inbound requests: if
  set, reuses that trace id instead of minting a new one. Stashes
  the trace id on the request context so handlers can attach
  application-level child spans.
- `validatord.chatCaller` forwards the header to chatd. Every chat
  hop in an iterate session lands as a child of the parent trace.

## Application-level spans

- `validator.IterateConfig` gains `Tracer` (optional callback).
  When wired, each iteration attempt emits one Langfuse span
  via `validator.AttemptSpan`:
    Name: iterate.attempt[N]
    Input: { iteration, model, provider, prompt }
    Output: { verdict, raw, error }
    Level: WARNING when verdict != accepted
- `validatord.iterTracer` is the production hook — bridges
  `validator.Tracer` → `langfuse.Client.Span`.
- `IterateRequest`/`IterateResponse`/`IterateFailure` gain
  `TraceID`; each `IterateAttempt` gains `SpanID`. The /v1/iterate
  caller can pivot from the JSON response straight into the
  Langfuse trace tree.

## What an operator sees post-cutover

  GET /v1/iterate {kind=fill, prompt=...} → Trace TR-1
    ├─ http.request span (from middleware)
    ├─ iterate.attempt[0] span (validator.Iterate emit)
    │     input: prompt+model
    │     output: { verdict: validation_failed, error: ..., raw }
    ├─ chatd /v1/chat call (X-Lakehouse-Trace-Id: TR-1)
    │     ├─ http.request span (chatd middleware)
    │     └─ chatd-internal spans (existing)
    ├─ iterate.attempt[1] span
    └─ ...

All in one Langfuse trace tree, not N+1 separate traces.

## Hallucinated-worker safety net is unchanged

The /v1/iterate flow's hard correctness gate is still
FillValidator + WorkerLookup. Phantom candidate IDs raise
ValidationError::Consistency which 422s and forces the iteration
loop to retry. The trace-id propagation is the OBSERVABILITY layer
on top — it makes the existing safety net's outcomes visible per-call,
not a replacement for it.

## Verification

- internal/validator: 4 new tests
  - TestIterate_TracerEmitsSpanPerAttempt — span/attempt count + SpanID
  - TestIterate_NoTraceIDSkipsTracer — no orphan spans without trace_id
  - TestIterate_ChatCallerReceivesTraceID — propagation contract
  - (existing iterate tests updated for new ChatCaller signature)
- internal/shared: 1 new test
  - TestLangfuseMiddleware_HonorsTraceIDHeader — cross-service linkage
- cmd/validatord: existing HTTP tests still PASS via the dual-shape
  UnmarshalJSON contract.
- validatord_smoke.sh: 5/5 PASS through gateway :3110 (unchanged).
- Full go test ./... green across 33 packages.

## Architecture invariant added

STATE_OF_PLAY "DO NOT RELITIGATE" gains a paragraph documenting
the X-Lakehouse-Trace-Id header contract + the iterate.attempt[N]
span emission. Future-Claude won't re-propose "wire trace-id
propagation" — the header IS the wiring.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-05-02 05:13:18 -05:00
co-authored by Claude Opus 4.7
parent afdeca80d9
commit d6d2fdf81f
7 changed files with 366 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cmd/validatord/main.go
+84 -2
View File
@@ -34,6 +34,7 @@ import (
"github.com/go-chi/chi/v5"
"git.agentview.dev/profit/golangLAKEHOUSE/internal/langfuse"
"git.agentview.dev/profit/golangLAKEHOUSE/internal/shared"
"git.agentview.dev/profit/golangLAKEHOUSE/internal/validator"
)
@@ -73,6 +74,11 @@ func main() {
DefaultMaxIterations: cfg.Validatord.DefaultMaxIterations,
DefaultMaxTokens: cfg.Validatord.DefaultMaxTokens,
},
// Same env loader the middleware uses — getting our own
// reference lets us emit child spans of the per-request trace.
// nil when Langfuse env isn't set; the iterate handler skips
// span emission gracefully in that case.
lf: shared.LoadLangfuseFromEnv(),
}
if err := shared.Run("validatord", cfg.Validatord.Bind, h.register, cfg.Auth); err != nil {
@@ -86,6 +92,12 @@ type handlers struct {
chatdURL string
chatClient *http.Client
iterCfg validator.IterateConfig
// lf is the Langfuse client (nil when unconfigured — best-effort
// posture matching the rest of the stack). Per-attempt iteration
// spans get emitted via lf when set. The HTTP middleware that
// creates the parent trace lives in shared.Run; this client lets
// validatord's handler emit application-level child spans.
lf *langfuse.Client
}
func (h *handlers) register(r chi.Router) {
@@ -224,6 +236,16 @@ func (h *handlers) handleIterate(w http.ResponseWriter, r *http.Request) {
return
}
// Pull the per-request trace id from the langfuse middleware. If
// the caller forwarded an upstream trace via X-Lakehouse-Trace-Id
// the middleware reuses that one; otherwise it minted a fresh trace
// at HTTP entry. Either way, we propagate it so chat hops nest
// under the same parent and operators can pivot from the iterate
// response's trace_id straight into the full Langfuse tree.
if req.TraceID == "" {
req.TraceID = shared.TraceIDFromCtx(r.Context())
}
chat := h.chatCaller()
validate := func(kind string, artifact map[string]any) (validator.Report, error) {
report, vErr, kindErr := h.runValidator(kind, artifact, req.Context)
@@ -239,7 +261,15 @@ func (h *handlers) handleIterate(w http.ResponseWriter, r *http.Request) {
return *report, nil
}
resp, fail, err := validator.Iterate(r.Context(), req, h.iterCfg, chat, validate)
// Wire per-attempt span emission when Langfuse is configured —
// each chat→validate cycle becomes a child span of the iterate
// trace, with prompt/raw/verdict captured for inspection.
cfg := h.iterCfg
if h.lf != nil {
cfg.Tracer = h.iterTracer(r.Context())
}
resp, fail, err := validator.Iterate(r.Context(), req, cfg, chat, validate)
if err != nil {
http.Error(w, err.Error(), http.StatusBadGateway)
return
@@ -254,8 +284,14 @@ func (h *handlers) handleIterate(w http.ResponseWriter, r *http.Request) {
// chatCaller wires the iteration loop to chatd via HTTP. Builds the
// chat.Request shape, posts to ${chatdURL}/chat, returns the content
// string (no choices wrapper — chatd's response is already flat).
//
// When traceID is non-empty, forwards it as the `X-Lakehouse-Trace-Id`
// header so chatd's middleware attaches its trace to the same
// Langfuse parent. Closes the multi-call observability gap: a
// /v1/iterate session shows in Langfuse as one trace with N child
// chat spans, instead of N+1 disconnected traces.
func (h *handlers) chatCaller() validator.ChatCaller {
return func(ctx context.Context, system, user, _, model string, temp *float64, maxTokens int) (string, error) {
return func(ctx context.Context, system, user, _, model string, temp *float64, maxTokens int, traceID string) (string, error) {
messages := make([]map[string]string, 0, 2)
if system != "" {
messages = append(messages, map[string]string{"role": "system", "content": system})
@@ -278,6 +314,9 @@ func (h *handlers) chatCaller() validator.ChatCaller {
return "", fmt.Errorf("build chat req: %w", err)
}
req.Header.Set("Content-Type", "application/json")
if traceID != "" {
req.Header.Set(shared.TraceIDHeader, traceID)
}
resp, err := h.chatClient.Do(req)
if err != nil {
return "", fmt.Errorf("chat hop: %w", err)
@@ -297,6 +336,49 @@ func (h *handlers) chatCaller() validator.ChatCaller {
}
}
// iterTracer adapts validator.Tracer → langfuse.Client.Span. Each
// iteration attempt becomes one Langfuse span on the parent trace
// with input={prompt, model, provider}, output={raw, verdict}.
// Operators reading Langfuse can scroll the iterate session and see
// the full retry chain — which prompt produced which raw, which
// validator verdict landed, which retry the model recovered on.
//
// Returns the span id so it lands in IterateAttempt.SpanID — the
// /v1/iterate response carries that id back to the caller for log
// correlation.
//
// Errors are silent (best-effort posture per package langfuse). A
// dropped span never blocks the iterate loop.
func (h *handlers) iterTracer(ctx context.Context) validator.Tracer {
return func(s validator.AttemptSpan) string {
level := "DEFAULT"
if s.Verdict.Kind != "accepted" {
level = "WARNING"
}
return h.lf.Span(ctx, langfuse.SpanInput{
TraceID: s.TraceID,
Name: fmt.Sprintf("iterate.attempt[%d]", s.Iteration),
Input: map[string]any{
"iteration": s.Iteration,
"model": s.Model,
"provider": s.Provider,
"prompt": trim(s.Prompt, 4000),
},
Output: map[string]any{
"verdict": s.Verdict.Kind,
"error": s.Verdict.Error,
"raw": trim(s.Raw, 4000),
},
Metadata: map[string]any{
"verdict_error": s.Verdict.Error,
},
StartTime: s.StartTime,
EndTime: s.EndTime,
Level: level,
})
}
}
func writeJSON(w http.ResponseWriter, status int, body any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)