profit/golangLAKEHOUSE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

materializer + replay ports + vectord substrate fix verified at scale

Browse Source 
Two threads landing together — the doc edits interleave so they ship
in a single commit.

1. **vectord substrate fix verified at original scale** (closes the
   2026-05-01 thread). Re-ran multitier 5min @ conc=50: 132,211
   scenarios at 438/sec, 6/6 classes at 0% failure (was 4/6 pre-fix).
   Throughput dropped 1,115 → 438/sec because previously-broken
   scenarios now do real HNSW Add work — honest cost of correctness.
   The fix (i.vectors side-store + safeGraphAdd recover wrappers +
   smallIndexRebuildThreshold=32 + saveTask coalescing) holds at the
   footprint that originally surfaced the bug.

2. **Materializer port** — internal/materializer + cmd/materializer +
   scripts/materializer_smoke.sh. Ports scripts/distillation/transforms.ts
   (12 transforms) + build_evidence_index.ts (idempotency, day-partition,
   receipt). On-wire JSON shape matches TS so Bun and Go runs are
   interchangeable. 14 tests green.

3. **Replay port** — internal/replay + cmd/replay +
   scripts/replay_smoke.sh. Ports scripts/distillation/replay.ts
   (retrieve → bundle → /v1/chat → validate → log). Closes audit-FULL
   phase 7 live invocation on the Go side. Both runtimes append to the
   same data/_kb/replay_runs.jsonl (schema=replay_run.v1). 14 tests green.

Side effect on internal/distillation/types.go: EvidenceRecord gained
prompt_tokens, completion_tokens, and metadata fields to mirror the TS
shape the materializer transforms produce.

STATE_OF_PLAY refreshed to 2026-05-02; ARCHITECTURE_COMPARISON decisions
tracker moves the materializer + replay items from _open_ to DONE and
adds the substrate-fix scale verification row.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-05-02 03:31:02 -05:00
co-authored by Claude Opus 4.7
parent 277884b5eb
commit 89ca72d471
26 changed files with 3961 additions and 124 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/vectord/index.go
+187 -86
View File
@@ -33,6 +33,23 @@ const (
DefaultEfSearch = 20
)
// smallIndexRebuildThreshold guards against coder/hnsw v0.6.1's
// degenerate-state nil-deref (graph.go:95 layerNode.search) which
// fires when the graph transitions through low-len states with a
// stale entry pointer. Below this threshold, Add and BatchAdd
// rebuild the entire graph from scratch — fresh graph + one
// variadic Add never exercises the buggy incremental path.
//
// Why 32: HNSW's value is sub-linear search at large N; at N<32 a
// rebuild's O(n) cost (snapshot ids + bulk Add) is negligible
// (~µs at 768-d). The boundary is intentionally above the small
// playbook-corpus regime (where multitier_100k surfaced the bug)
// but well below realistic working-set indexes.
//
// The recover() guard in BatchAdd remains as belt-and-suspenders
// for any incremental-path edge cases past the threshold.
const smallIndexRebuildThreshold = 32
// IndexParams describes one vector index. Once an Index is built,
// these are fixed — changing M / dimension / distance requires a
// rebuild.
@@ -55,21 +72,30 @@ type Result struct {
Metadata json.RawMessage `json:"metadata,omitempty"`
}
// Index wraps a coder/hnsw graph plus a side map of opaque JSON
// metadata per ID. Concurrency: read-heavy via Search (read-lock);
// Add and Delete take the write lock.
// Index wraps a coder/hnsw graph plus side maps of opaque JSON
// metadata and raw vectors per ID. Concurrency: read-heavy via
// Search (read-lock); Add and Delete take the write lock.
//
// Why we keep vectors in a side map (i.vectors) in addition to the
// graph: coder/hnsw v0.6.1 has a known bug where the graph
// transitions through degenerate states after Delete cycles, and
// later operations (Add / Lookup) can panic with nil-deref. The
// side map is independent of graph state, so the rebuild path can
// always reconstruct a clean graph even if the current one is
// corrupted. Memory cost is ~2x for vectors (also held in graph),
// which is acceptable for the safety it buys. Verified necessary
// 2026-05-01 multitier_100k where the bug fired at len=40.
type Index struct {
params IndexParams
g *hnsw.Graph[string]
meta map[string]json.RawMessage
// ids is the canonical ID set (a value-less map used as a set).
// Maintained alongside i.g and i.meta in Add/Delete/resetGraph
// so IDs() can enumerate without depending on the meta map's
// sparse-on-nil-meta semantics. Underpins OPEN #1's merge
// endpoint — necessary because two-tier callers
// (multi_coord_stress et al.) sometimes Add with nil meta.
ids map[string]struct{}
mu sync.RWMutex
// vectors is the panic-safe source of truth — every successful
// Add stores the vector here, every Delete removes it, and
// rebuildGraphLocked reads from this map (not i.g.Lookup) so
// it tolerates a corrupted graph. Map keys are also the
// canonical ID set (replaces the prior i.ids map).
vectors map[string][]float32
mu sync.RWMutex
}
// Errors surfaced to HTTP handlers. Sentinel-based so the wire
@@ -110,10 +136,10 @@ func NewIndex(p IndexParams) (*Index, error) {
// is a G2 concern when we have real tuning data.
return &Index{
params: p,
g: g,
meta: make(map[string]json.RawMessage),
ids: make(map[string]struct{}),
params: p,
g: g,
meta: make(map[string]json.RawMessage),
vectors: make(map[string][]float32),
}, nil
}
@@ -133,10 +159,14 @@ func distanceFn(name string) (hnsw.DistanceFunc, error) {
func (i *Index) Params() IndexParams { return i.params }
// Len returns the number of vectors currently in the index.
//
// Reads from i.vectors (the panic-safe source of truth) rather
// than i.g.Len() — the latter can drift past Len during a corrupted
// graph state. i.vectors only changes on successful Add/Delete.
func (i *Index) Len() int {
i.mu.RLock()
defer i.mu.RUnlock()
return i.g.Len()
return len(i.vectors)
}
// IDs returns a snapshot of every ID currently stored in the index.
@@ -145,16 +175,15 @@ func (i *Index) Len() int {
// (OPEN #1: periodic fresh→main index merge — drains the fresh
// corpus into the main one when it crosses the operational ceiling).
//
// Source of truth: the i.ids tracker, NOT the meta map. The meta
// map intentionally stays sparse (only items with explicit
// metadata appear there, per the K-B1 nil-vs-{} distinction). Using
// meta as the ID set would silently miss items added with nil
// metadata.
// Source of truth: the i.vectors keyset. The meta map stays sparse
// (only items with explicit metadata appear there, per the K-B1
// nil-vs-{} distinction); using meta as the ID set would silently
// miss items added with nil metadata.
func (i *Index) IDs() []string {
i.mu.RLock()
defer i.mu.RUnlock()
out := make([]string, 0, len(i.ids))
for id := range i.ids {
out := make([]string, 0, len(i.vectors))
for id := range i.vectors {
out = append(out, id)
}
return out
@@ -191,23 +220,38 @@ func (i *Index) Add(id string, vec []float32, meta json.RawMessage) error {
}
i.mu.Lock()
defer i.mu.Unlock()
// coder/hnsw has two sharp edges on re-add:
// 1. Add of an existing key panics with "node not added"
// (length-invariant fires because internal delete+re-add
// doesn't change Len). Pre-Delete fixes this for n>1.
// 2. Delete of the LAST node leaves layers[0] non-empty but
// entryless; the next Add SIGSEGVs in Dims() because
// entry().Value is nil. We rebuild the graph in that case.
_, exists := i.g.Lookup(id)
if exists {
if i.g.Len() == 1 {
i.resetGraphLocked()
} else {
i.g.Delete(id)
// Re-add: drop existing graph entry AND side-store entry before
// the new Add. Without removing from i.vectors, the rebuild path
// below would see both old and new entries and double-add.
// safeGraphDelete tolerates a corrupted graph; i.vectors is
// authoritative regardless.
if _, exists := i.vectors[id]; exists {
_ = safeGraphDelete(i.g, id)
delete(i.vectors, id)
}
newNode := hnsw.MakeNode(id, vec)
postLen := len(i.vectors) + 1
addOK := false
if postLen <= smallIndexRebuildThreshold {
i.rebuildGraphLocked([]hnsw.Node[string]{newNode})
addOK = true
} else {
// Warm path: try incremental Add. If the graph is in a
// degenerate state from a prior Delete cycle, this panics;
// we recover and rebuild from the panic-safe i.vectors map.
addOK = safeGraphAdd(i.g, newNode)
if !addOK {
i.rebuildGraphLocked([]hnsw.Node[string]{newNode})
addOK = true
}
}
i.g.Add(hnsw.MakeNode(id, vec))
i.ids[id] = struct{}{}
if !addOK {
return errors.New("vectord: hnsw add failed even after rebuild — should never happen")
}
// Commit to the side stores after the graph mutation succeeded.
out := make([]float32, len(vec))
copy(out, vec)
i.vectors[id] = out
if meta != nil {
// Per scrum K-B1 (Kimi): only OVERWRITE on explicit non-nil.
// nil = "leave existing meta alone" (upsert). To clear, the
@@ -217,17 +261,59 @@ func (i *Index) Add(id string, vec []float32, meta json.RawMessage) error {
return nil
}
// resetGraphLocked recreates the underlying coder/hnsw Graph with
// the same params. Caller MUST hold i.mu (write-lock). Used to
// dodge the library's "delete the last node, then segfault on
// next Add" bug — see Add for details. Metadata map is preserved
// because the only entry it could affect is the one being
// re-added, which Add overwrites.
func (i *Index) resetGraphLocked() {
// safeGraphAdd wraps coder/hnsw's variadic Graph.Add with a
// recover() so v0.6.1's degenerate-state nil-deref returns false
// instead of crashing the caller. Caller is expected to fall back
// to rebuildGraphLocked on false.
func safeGraphAdd(g *hnsw.Graph[string], nodes ...hnsw.Node[string]) (ok bool) {
defer func() {
if r := recover(); r != nil {
ok = false
}
}()
g.Add(nodes...)
return true
}
// safeGraphDelete wraps Graph.Delete with recover for the same
// reason — Delete can also touch corrupted layer state.
func safeGraphDelete(g *hnsw.Graph[string], id string) (ok bool) {
defer func() {
if r := recover(); r != nil {
ok = false
}
}()
return g.Delete(id)
}
// rebuildGraphLocked replaces i.g with a fresh graph containing
// the current items (snapshotted from the panic-safe i.vectors
// map) plus the supplied extras, in one bulk Add into a freshly-
// created graph. Caller MUST hold the write lock.
//
// Independence from i.g state is the load-bearing property — even
// if i.g is corrupted from a prior coder/hnsw v0.6.1 panic, this
// rebuild produces a clean graph because i.vectors is maintained
// only on successful Add/Delete.
//
// Caller MUST ensure that any extra IDs already present in
// i.vectors have been removed first (otherwise the bulk Add will
// see duplicate IDs and panic).
func (i *Index) rebuildGraphLocked(extras []hnsw.Node[string]) {
g := hnsw.NewGraph[string]()
g.M = i.params.M
g.EfSearch = i.params.EfSearch
g.Distance = i.g.Distance
nodes := make([]hnsw.Node[string], 0, len(i.vectors)+len(extras))
for id, vec := range i.vectors {
nodes = append(nodes, hnsw.MakeNode(id, vec))
}
nodes = append(nodes, extras...)
if len(nodes) > 0 {
g.Add(nodes...)
}
i.g = g
}
@@ -296,17 +382,15 @@ func (i *Index) BatchAdd(items []BatchItem) error {
i.mu.Lock()
defer i.mu.Unlock()
// Pre-pass: drop any existing IDs so coder/hnsw's variadic Add
// never sees a re-add. Same library-quirk handling as single
// Add — Len()==1 needs a full graph reset because Delete of the
// last node leaves layers[0] entryless.
// Pre-pass: drop any existing IDs from BOTH the graph and the
// side-store map so the rebuild snapshot doesn't double-add and
// the warm path's variadic Add never sees a re-add. Graph Delete
// is wrapped in safeGraphDelete because corrupted graphs can also
// panic on Delete; the side store remains authoritative.
for _, it := range items {
if _, exists := i.g.Lookup(it.ID); exists {
if i.g.Len() == 1 {
i.resetGraphLocked()
} else {
i.g.Delete(it.ID)
}
if _, exists := i.vectors[it.ID]; exists {
_ = safeGraphDelete(i.g, it.ID)
delete(i.vectors, it.ID)
}
}
@@ -314,27 +398,26 @@ func (i *Index) BatchAdd(items []BatchItem) error {
for j, it := range items {
nodes[j] = hnsw.MakeNode(it.ID, it.Vector)
}
// coder/hnsw v0.6.1 has a known nil-deref in layerNode.search at
// graph.go:95 when the graph transitions through degenerate
// states (len=0/1 with stale entry from a prior Delete cycle).
// Wrap with recover so a panic becomes an error rather than
// killing the request handler. Surfaced under sustained
// playbook_record load (multitier test 2026-05-01); operator
// recovery is `DELETE /vectors/index/<name>` then re-record.
if addErr := func() (err error) {
defer func() {
if r := recover(); r != nil {
err = fmt.Errorf("hnsw add panic (coder/hnsw v0.6.1 small-index bug — DELETE the index to recover): %v", r)
}
}()
i.g.Add(nodes...)
return nil
}(); addErr != nil {
return addErr
// Below threshold: rebuild from scratch unconditionally — fresh
// graph + one bulk Add never exercises v0.6.1's degenerate-state
// path. At/above threshold: try warm incremental Add, fall back
// to rebuild on panic. The rebuild always succeeds because
// i.vectors is independent of graph state.
postLen := len(i.vectors) + len(nodes)
if postLen <= smallIndexRebuildThreshold {
i.rebuildGraphLocked(nodes)
} else {
if !safeGraphAdd(i.g, nodes...) {
i.rebuildGraphLocked(nodes)
}
}
// Commit to side stores after the graph is in good shape.
for _, it := range items {
i.ids[it.ID] = struct{}{}
out := make([]float32, len(it.Vector))
copy(out, it.Vector)
i.vectors[it.ID] = out
if it.Metadata != nil {
i.meta[it.ID] = it.Metadata
}
@@ -374,12 +457,22 @@ func dedupBatchLastWins(items []BatchItem) []BatchItem {
}
// Delete removes id from the index. Returns true if present.
//
// The side store i.vectors is the authority on presence; the graph
// Delete is best-effort (can panic on corrupted state, recovered
// via safeGraphDelete). The side store always reflects the
// post-Delete truth so the next rebuild produces a clean graph.
func (i *Index) Delete(id string) bool {
i.mu.Lock()
defer i.mu.Unlock()
_, present := i.vectors[id]
if !present {
return false
}
delete(i.meta, id)
delete(i.ids, id)
return i.g.Delete(id)
delete(i.vectors, id)
_ = safeGraphDelete(i.g, id)
return true
}
// Search returns the k nearest neighbors of query, sorted
@@ -456,9 +549,9 @@ func (i *Index) Encode(envelopeW, graphW io.Writer) error {
defer i.mu.RUnlock()
// v2: serialize the canonical ID set explicitly so DecodeIndex
// can restore i.ids without depending on meta-key inference.
idList := make([]string, 0, len(i.ids))
for id := range i.ids {
// can restore i.vectors without depending on meta-key inference.
idList := make([]string, 0, len(i.vectors))
for id := range i.vectors {
idList = append(idList, id)
}
env := IndexEnvelope{
@@ -501,19 +594,27 @@ func DecodeIndex(envelopeR, graphR io.Reader) (*Index, error) {
if env.Metadata != nil {
idx.meta = env.Metadata
}
// v2: explicit IDs field is the canonical source. v1 fallback:
// derive from meta keys, accepting that nil-meta items will be
// invisible to IDs()/merge until they get re-Add'd. Closes the
// scrum post_role_gate_v1 convergent finding (Opus + Kimi).
// Reconstruct i.vectors from the imported graph. Source of IDs:
// v2 envelope's explicit IDs slice (canonical), or v1 fallback
// via the meta keys. We then call i.g.Lookup on each ID to
// recover the vector — Lookup on a freshly Imported graph is
// safe (no degenerate state from prior Delete cycles).
var idSource []string
if env.Version >= 2 && env.IDs != nil {
for _, id := range env.IDs {
idx.ids[id] = struct{}{}
}
idSource = env.IDs
} else {
// v1 backward-compat path. Old envelopes don't carry ids
// explicitly; the metadata keyset is the best signal we have.
idSource = make([]string, 0, len(idx.meta))
for id := range idx.meta {
idx.ids[id] = struct{}{}
idSource = append(idSource, id)
}
}
for _, id := range idSource {
if vec, ok := idx.g.Lookup(id); ok {
out := make([]float32, len(vec))
copy(out, vec)
idx.vectors[id] = out
}
}
return idx, nil