G1P: vectord persistence to storaged + scrum (3 fixes incl. 3-way convergent)

Adds optional persistence to vectord (G1's HNSW vector search). Single-
file framed format per index — eliminates the torn-write class that
the 3-way convergent scrum finding identified:

  _vectors/<name>.lhv1  — single binary blob:
      [4 bytes magic "LHV1"]
      [4 bytes envelope_len uint32 BE]
      [envelope bytes — JSON params + metadata + version]
      [graph bytes — raw hnsw.Graph.Export]

Pre-extraction: internal/catalogd/store_client.go → internal/storeclient/
shared package, since both catalogd and vectord need it. Same pattern as
the pre-D5 catalogclient extraction.

Optional via [vectord].storaged_url config (empty = ephemeral mode).
On startup: List + Load each persisted index. After Create / batch Add /
DELETE: Save (or Delete from storaged). Save failures are logged-not-
fatal — in-memory state is the source of truth in flight.

Acceptance smoke G1P 8/8 PASS — kill+restart preserves state, post-
restart search returns dist=0 (graph round-trips exactly), DELETE
removes the file, post-delete restart shows count=0.

All 8 smokes (D1-D6 + G1 + G1P) PASS deterministically. The g1_smoke
gained scripts/g1_smoke.toml that disables persistence so the
in-memory API test stays decoupled from any rehydrate-from-storaged
state contamination.

Cross-lineage scrum on shipped code:
  - Opus 4.7 (opencode):                     1 BLOCK + 5 WARN + 3 INFO
  - Kimi K2-0905 (openrouter):               1 BLOCK + 2 WARN
  - Qwen3-coder (openrouter):                2 BLOCK + 2 WARN + 1 INFO

Fixed (3 — 1 convergent + 2 single-reviewer):
  C1 (Opus + Kimi + Qwen 3-WAY CONVERGENT WARN): Save was non-atomic
    across two PUTs — envelope-succeeds + graph-fails left a half-
    saved index that passed the "both present" List filter and
    silently mismatched metadata against vectors on Load.
    Fix: collapse to single framed file (no torn-write window
    possible).
  O-B1 (Opus BLOCK): isNotFound substring-matched "key not found"
    against the wrapped error message — brittle, any 5xx body
    containing that text would silently misclassify as missing.
    Fix: errors.Is(err, storeclient.ErrKeyNotFound).
  O-I3 (Opus INFO): handleAdd pre-validation only covered id+dim;
    NaN/Inf/zero-norm could still fail mid-batch leaving partial
    commits. Fix: extend pre-validation to call ValidateVector
    (newly exported) per item before any commit.

Dismissed (3 false positives):
  K-B1 + Q-B1 ("safeKey double-escapes %2F segments") — false
    convergent. Wire-protocol escape is decoded by storaged's chi
    router on the way in; on-disk key is the original literal.
    %2F round-trips correctly through PathEscape → URL → chi decode
    → S3 key.
  Q-B2 ("List vulnerable to race conditions") — vectord is single-
    process; no concurrent Save against List in the same vectord.

Deferred (3): rehydrate per-index timeout (G2+ multi-index scale),
saveAfter request ctx (matches G0 timeout deferral), Encode RLock
during slow writer (documented as buffer-only API).

The C1 finding is the strongest signal of the cross-lineage filter:
three independent reviewers all flagged the same torn-write hazard.
Single-file framing eliminates the class — there's now no Persistor
state where envelope and graph can disagree.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-29 01:33:23 -05:00
co-authored by Claude Opus 4.7
parent b8c072cf0b
commit 8b92518d21
14 changed files with 859 additions and 57 deletions
+60
View File
@@ -1,6 +1,7 @@
package vectord
import (
"bytes"
"encoding/json"
"errors"
"fmt"
@@ -129,6 +130,65 @@ func TestIndex_ConcurrentSearchAdd(t *testing.T) {
wg.Wait()
}
func TestEncodeDecode_RoundTrip(t *testing.T) {
const n = 16
src, _ := NewIndex(IndexParams{Name: "x", Dimension: n, Distance: DistanceCosine})
mkVec := func(i int) []float32 {
// Each vector is a unique unit vector along axis (i mod n) with
// a tiny perturbation on a different axis — recall=1 is robust
// without colliding under cosine.
v := make([]float32, n)
v[i%n] = 1.0
v[(i+1)%n] = 0.001
return v
}
for i := 0; i < n; i++ {
meta := json.RawMessage(fmt.Sprintf(`{"row":%d}`, i))
if err := src.Add(fmt.Sprintf("id-%02d", i), mkVec(i), meta); err != nil {
t.Fatal(err)
}
}
var envBuf, graphBuf bytes.Buffer
if err := src.Encode(&envBuf, &graphBuf); err != nil {
t.Fatalf("Encode: %v", err)
}
dst, err := DecodeIndex(&envBuf, &graphBuf)
if err != nil {
t.Fatalf("DecodeIndex: %v", err)
}
if dst.Len() != src.Len() {
t.Errorf("Len: src=%d dst=%d", src.Len(), dst.Len())
}
if dst.Params() != src.Params() {
t.Errorf("Params: src=%+v dst=%+v", src.Params(), dst.Params())
}
for i := 0; i < n; i++ {
hits, err := dst.Search(mkVec(i), 1)
if err != nil {
t.Fatal(err)
}
want := fmt.Sprintf("id-%02d", i)
if len(hits) == 0 || hits[0].ID != want {
t.Errorf("Search after decode: id-%d → %v, want %s", i, hits, want)
continue
}
wantMeta := fmt.Sprintf(`{"row":%d}`, i)
if string(hits[0].Metadata) != wantMeta {
t.Errorf("metadata after decode: got %q, want %q", hits[0].Metadata, wantMeta)
}
}
}
func TestDecodeIndex_VersionMismatch(t *testing.T) {
bad := bytes.NewBufferString(`{"version":999,"params":{"name":"x","dimension":4}}`)
_, err := DecodeIndex(bad, bytes.NewReader(nil))
if !errors.Is(err, ErrVersionMismatch) {
t.Errorf("expected ErrVersionMismatch, got %v", err)
}
}
func TestRegistry_CreateGetDelete(t *testing.T) {
r := NewRegistry()
idx, err := r.Create(IndexParams{Name: "workers", Dimension: 4})