materializer + replay ports + vectord substrate fix verified at scale

Two threads landing together — the doc edits interleave so they ship
in a single commit.

1. **vectord substrate fix verified at original scale** (closes the
   2026-05-01 thread). Re-ran multitier 5min @ conc=50: 132,211
   scenarios at 438/sec, 6/6 classes at 0% failure (was 4/6 pre-fix).
   Throughput dropped 1,115 → 438/sec because previously-broken
   scenarios now do real HNSW Add work — honest cost of correctness.
   The fix (i.vectors side-store + safeGraphAdd recover wrappers +
   smallIndexRebuildThreshold=32 + saveTask coalescing) holds at the
   footprint that originally surfaced the bug.

2. **Materializer port** — internal/materializer + cmd/materializer +
   scripts/materializer_smoke.sh. Ports scripts/distillation/transforms.ts
   (12 transforms) + build_evidence_index.ts (idempotency, day-partition,
   receipt). On-wire JSON shape matches TS so Bun and Go runs are
   interchangeable. 14 tests green.

3. **Replay port** — internal/replay + cmd/replay +
   scripts/replay_smoke.sh. Ports scripts/distillation/replay.ts
   (retrieve → bundle → /v1/chat → validate → log). Closes audit-FULL
   phase 7 live invocation on the Go side. Both runtimes append to the
   same data/_kb/replay_runs.jsonl (schema=replay_run.v1). 14 tests green.

Side effect on internal/distillation/types.go: EvidenceRecord gained
prompt_tokens, completion_tokens, and metadata fields to mirror the TS
shape the materializer transforms produce.

STATE_OF_PLAY refreshed to 2026-05-02; ARCHITECTURE_COMPARISON decisions
tracker moves the materializer + replay items from _open_ to DONE and
adds the substrate-fix scale verification row.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

cmd/vectord/main.go

@@ -17,6 +17,7 @@ import (
 	"os"
 	"strconv"
 	"strings"
+	"sync"
 	"time"
 
 	"github.com/go-chi/chi/v5"
@@ -71,6 +72,73 @@ func main() {
 type handlers struct {
 	reg     *vectord.Registry
 	persist *vectord.Persistor // nil when persistence is disabled
+
+	// saversMu guards lazy initialization of per-index save tasks.
+	// Each task coalesces synchronous Save calls into single-flight
+	// async saves so high-write-rate indexes (playbook_memory under
+	// multitier_100k load) don't pay one MinIO PUT per Add. See the
+	// saveTask docstring for the coalescing semantics.
+	saversMu sync.Mutex
+	savers   map[string]*saveTask
+}
+
+// saveTask coalesces saves for one index into a single-flight async
+// goroutine. While a save is in-flight, additional triggers mark
+// "pending" — the in-flight goroutine reruns the save after it
+// finishes, collapsing N concurrent triggers into at most 2 saves
+// (the current in-flight + one catch-up).
+//
+// Why: pre-2026-05-01 each successful Add called Persistor.Save
+// synchronously inside the request handler. For playbook_memory at
+// 1900-entry / 768-d, Encode + MinIO PUT cost 100-300ms. With 50
+// concurrent writers, end-to-end Add latency hit 2-2.5s purely from
+// save serialization (Save takes the index RLock for Encode, which
+// blocks new Adds taking the Lock).
+//
+// Trade-off: RPO. Add now returns OK before the save completes, so
+// a crash can lose up to ~1 save's worth of data. Acceptable for
+// the playbook-memory shape (learning loop — lost trace re-recorded
+// on next run) and consistent with ADR-005's fail-open posture.
+type saveTask struct {
+	mu       sync.Mutex
+	inflight bool
+	pending  bool
+}
+
+// trigger schedules a save. If a save is already in-flight, marks
+// pending and returns. If none in-flight, starts a goroutine that
+// runs save and any queued pending saves.
+//
+// save is the actual save operation (parameterized for testability).
+// Errors are logged via slog and not returned — same fail-open
+// posture as the prior synchronous saveAfter.
+func (s *saveTask) trigger(save func() error) {
+	s.mu.Lock()
+	if s.inflight {
+		s.pending = true
+		s.mu.Unlock()
+		return
+	}
+	s.inflight = true
+	s.mu.Unlock()
+
+	go func() {
+		for {
+			if err := save(); err != nil {
+				slog.Warn("persist save", "err", err)
+			}
+			s.mu.Lock()
+			if !s.pending {
+				s.inflight = false
+				s.mu.Unlock()
+				return
+			}
+			s.pending = false
+			s.mu.Unlock()
+			// Loop: re-run save to capture changes that arrived
+			// while we were saving.
+		}
+	}()
 }
 
 // rehydrate enumerates persisted indexes and loads each into the
@@ -103,19 +171,38 @@ func (h *handlers) rehydrate(ctx context.Context) (int, error) {
 	return loaded, nil
 }
 
-// saveAfter is the post-write persistence hook. Logs-not-fatal:
-// in-memory state is the source of truth in flight; a failed save
-// gets re-attempted on the next mutation, and the operator log
-// shows the storaged outage.
+// saveAfter triggers a coalesced async persistence for the index.
+// In-memory state is the source of truth in flight; a failed save
+// re-runs on the next mutation, and the operator log shows the
+// storaged outage.
+//
+// Coalescing semantics (added 2026-05-01 after multitier_100k
+// follow-up): rapid concurrent writes collapse into at most two
+// MinIO PUTs per index (current + one catch-up), instead of one
+// per Add. See the saveTask docstring.
 func (h *handlers) saveAfter(idx *vectord.Index) {
 	if h.persist == nil {
 		return
 	}
-	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
-	defer cancel()
-	if err := h.persist.Save(ctx, idx); err != nil {
-		slog.Warn("persist save", "name", idx.Params().Name, "err", err)
+	name := idx.Params().Name
+	h.saversMu.Lock()
+	if h.savers == nil {
+		h.savers = make(map[string]*saveTask)
 	}
+	s, ok := h.savers[name]
+	if !ok {
+		s = &saveTask{}
+		h.savers[name] = s
+	}
+	h.saversMu.Unlock()
+	s.trigger(func() error {
+		ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+		defer cancel()
+		if err := h.persist.Save(ctx, idx); err != nil {
+			return err
+		}
+		return nil
+	})
 }
 
 // deleteAfter mirrors saveAfter for the Delete path.