profit/golangLAKEHOUSE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

G1P: vectord persistence to storaged + scrum (3 fixes incl. 3-way convergent)

Browse Source 
Adds optional persistence to vectord (G1's HNSW vector search). Single-
file framed format per index — eliminates the torn-write class that
the 3-way convergent scrum finding identified:

  _vectors/<name>.lhv1  — single binary blob:
      [4 bytes magic "LHV1"]
      [4 bytes envelope_len uint32 BE]
      [envelope bytes — JSON params + metadata + version]
      [graph bytes — raw hnsw.Graph.Export]

Pre-extraction: internal/catalogd/store_client.go → internal/storeclient/
shared package, since both catalogd and vectord need it. Same pattern as
the pre-D5 catalogclient extraction.

Optional via [vectord].storaged_url config (empty = ephemeral mode).
On startup: List + Load each persisted index. After Create / batch Add /
DELETE: Save (or Delete from storaged). Save failures are logged-not-
fatal — in-memory state is the source of truth in flight.

Acceptance smoke G1P 8/8 PASS — kill+restart preserves state, post-
restart search returns dist=0 (graph round-trips exactly), DELETE
removes the file, post-delete restart shows count=0.

All 8 smokes (D1-D6 + G1 + G1P) PASS deterministically. The g1_smoke
gained scripts/g1_smoke.toml that disables persistence so the
in-memory API test stays decoupled from any rehydrate-from-storaged
state contamination.

Cross-lineage scrum on shipped code:
  - Opus 4.7 (opencode):                     1 BLOCK + 5 WARN + 3 INFO
  - Kimi K2-0905 (openrouter):               1 BLOCK + 2 WARN
  - Qwen3-coder (openrouter):                2 BLOCK + 2 WARN + 1 INFO

Fixed (3 — 1 convergent + 2 single-reviewer):
  C1 (Opus + Kimi + Qwen 3-WAY CONVERGENT WARN): Save was non-atomic
    across two PUTs — envelope-succeeds + graph-fails left a half-
    saved index that passed the "both present" List filter and
    silently mismatched metadata against vectors on Load.
    Fix: collapse to single framed file (no torn-write window
    possible).
  O-B1 (Opus BLOCK): isNotFound substring-matched "key not found"
    against the wrapped error message — brittle, any 5xx body
    containing that text would silently misclassify as missing.
    Fix: errors.Is(err, storeclient.ErrKeyNotFound).
  O-I3 (Opus INFO): handleAdd pre-validation only covered id+dim;
    NaN/Inf/zero-norm could still fail mid-batch leaving partial
    commits. Fix: extend pre-validation to call ValidateVector
    (newly exported) per item before any commit.

Dismissed (3 false positives):
  K-B1 + Q-B1 ("safeKey double-escapes %2F segments") — false
    convergent. Wire-protocol escape is decoded by storaged's chi
    router on the way in; on-disk key is the original literal.
    %2F round-trips correctly through PathEscape → URL → chi decode
    → S3 key.
  Q-B2 ("List vulnerable to race conditions") — vectord is single-
    process; no concurrent Save against List in the same vectord.

Deferred (3): rehydrate per-index timeout (G2+ multi-index scale),
saveAfter request ctx (matches G0 timeout deferral), Encode RLock
during slow writer (documented as buffer-only API).

The C1 finding is the strongest signal of the cross-lineage filter:
three independent reviewers all flagged the same torn-write hazard.
Single-file framing eliminates the class — there's now no Persistor
state where envelope and graph can disagree.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-29 01:33:23 -05:00
co-authored by Claude Opus 4.7
parent b8c072cf0b
commit 8b92518d21
14 changed files with 859 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/g1_smoke.sh
+8 -7
View File
@@ -46,10 +46,10 @@ poll_health() {
}
echo "[g1-smoke] launching vectord → gateway..."
./bin/vectord > /tmp/vectord.log 2>&1 &
./bin/vectord -config scripts/g1_smoke.toml > /tmp/vectord.log 2>&1 &
PIDS+=($!)
poll_health 3215 || { echo "vectord failed"; tail /tmp/vectord.log; exit 1; }
./bin/gateway > /tmp/gateway.log 2>&1 &
./bin/gateway -config scripts/g1_smoke.toml > /tmp/gateway.log 2>&1 &
PIDS+=($!)
poll_health 3110 || { echo "gateway failed"; tail /tmp/gateway.log; exit 1; }
@@ -81,14 +81,15 @@ echo "[g1-smoke] add batch of 200 vectors:"
# Build a batch payload with 200 items. ID encodes index; vector
# is a deterministic spread so we can assert recall on a known id.
python3 - <<EOF > "$TMP/batch.json"
import json, math
import json
items = []
for i in range(200):
# Each vector is unit-norm with one dominant axis cycling through 8 dims.
# Each vector is unit-norm-ish with one dominant axis cycling
# through DIM dims, and a UNIQUE perturbation derived from i itself
# (no modulo — would create collisions when 200/DIM rolls over).
v = [0.0] * $DIM
v[i % $DIM] = 1.0
# Add a tiny perturbation per id so vectors are distinct.
v[(i+1) % $DIM] = (i % 17) * 0.01
v[(i+1) % $DIM] = i * 0.0001 # unique per i
items.append({"id": f"w-{i:03d}", "vector": v, "metadata": {"row": i}})
print(json.dumps({"items": items}))
EOF
@@ -110,7 +111,7 @@ import json
i = 42
v = [0.0] * $DIM
v[i % $DIM] = 1.0
v[(i+1) % $DIM] = (i % 17) * 0.01
v[(i+1) % $DIM] = i * 0.0001 # match the unique-per-i perturbation
print(json.dumps({"vector": v, "k": 3}))
EOF
SEARCH_RESP="$(curl -sS -X POST "http://127.0.0.1:3110/v1/vectors/index/$NAME/search" \
scripts/g1_smoke.toml
+24
View File
@@ -0,0 +1,24 @@
# g1_smoke override — disables vectord persistence so the in-memory
# API test stays pure (no rehydrate-from-storaged contamination).
# g1p_smoke covers the persistence path.
[gateway]
bind = "127.0.0.1:3110"
storaged_url = "http://127.0.0.1:3211"
catalogd_url = "http://127.0.0.1:3212"
ingestd_url = "http://127.0.0.1:3213"
queryd_url = "http://127.0.0.1:3214"
vectord_url = "http://127.0.0.1:3215"
[storaged]
bind = "127.0.0.1:3211"
[vectord]
bind = "127.0.0.1:3215"
storaged_url = "" # disable persistence — pure in-memory for this smoke
[s3]
endpoint = "http://localhost:9000"
region = "us-east-1"
bucket = "lakehouse-go-primary"
use_path_style = true
scripts/g1p_smoke.sh
Executable
+167
View File
@@ -0,0 +1,167 @@
#!/usr/bin/env bash
# G1P smoke — vectord persistence across restart.
#
# Validates:
# - Create index + add 50 vectors → Save fires (storaged shows
# _vectors/persist_demo.json AND _vectors/persist_demo.hnsw)
# - Search w-001 → recall=1 with distance ≈ 0
# - Kill vectord → restart → initial Refresh enumerates persisted
# indexes, Loads each back into the registry
# - Same search post-restart → still recall=1 (state survived)
# - DELETE → both files removed from storaged → restart → index gone
#
# Requires storaged + vectord both up.
#
# Usage: ./scripts/g1p_smoke.sh
set -euo pipefail
cd "$(dirname "$0")/.."
export PATH="$PATH:/usr/local/go/bin"
echo "[g1p-smoke] building storaged + vectord + gateway..."
go build -o bin/ ./cmd/storaged ./cmd/vectord ./cmd/gateway
pkill -f "bin/(storaged|vectord|gateway)" 2>/dev/null || true
sleep 0.3
PIDS=()
TMP="$(mktemp -d)"
cleanup() {
echo "[g1p-smoke] cleanup"
for p in "${PIDS[@]}"; do [ -n "$p" ] && kill "$p" 2>/dev/null || true; done
rm -rf "$TMP"
}
trap cleanup EXIT INT TERM
poll_health() {
local port="$1" deadline=$(($(date +%s) + 5))
while [ "$(date +%s)" -lt "$deadline" ]; do
if curl -sS --max-time 1 "http://127.0.0.1:$port/health" >/dev/null 2>&1; then return 0; fi
sleep 0.05
done
return 1
}
NAME="persist_demo"
DIM=8
echo "[g1p-smoke] launching storaged..."
./bin/storaged > /tmp/storaged.log 2>&1 &
PIDS+=($!)
poll_health 3211 || { echo "storaged failed"; tail /tmp/storaged.log; exit 1; }
# Clean any prior persisted state for this index name.
curl -sS -o /dev/null -X DELETE "http://127.0.0.1:3211/storage/delete/_vectors/$NAME.lhv1" || true
launch_vectord() {
./bin/vectord > /tmp/vectord.log 2>&1 &
VECTORD_PID=$!
PIDS+=($VECTORD_PID)
poll_health 3215 || { echo "vectord failed"; tail /tmp/vectord.log; return 1; }
}
launch_gateway() {
./bin/gateway > /tmp/gateway.log 2>&1 &
PIDS+=($!)
poll_health 3110 || { echo "gateway failed"; tail /tmp/gateway.log; return 1; }
}
echo "[g1p-smoke] launching vectord (round 1) → gateway..."
launch_vectord
launch_gateway
FAILED=0
echo "[g1p-smoke] create index + add 50 vectors:"
HTTP="$(curl -sS -o /dev/null -w '%{http_code}' -X POST http://127.0.0.1:3110/v1/vectors/index \
-H 'Content-Type: application/json' \
-d "{\"name\":\"$NAME\",\"dimension\":$DIM,\"distance\":\"cosine\"}")"
if [ "$HTTP" != "201" ]; then echo " ✗ create → $HTTP"; FAILED=1; fi
# Build a 50-vector batch.
python3 - <<EOF > "$TMP/batch.json"
import json
items = []
for i in range(50):
v = [0.0] * $DIM
v[i % $DIM] = 1.0
v[(i+1) % $DIM] = (i % 13) * 0.01
items.append({"id": f"w-{i:03d}", "vector": v, "metadata": {"row": i}})
print(json.dumps({"items": items}))
EOF
LEN="$(curl -sS -X POST "http://127.0.0.1:3110/v1/vectors/index/$NAME/add" \
-H 'Content-Type: application/json' \
-d @"$TMP/batch.json" | jq -r '.length')"
if [ "$LEN" = "50" ]; then echo " ✓ added 50 → length=50"; else echo " ✗ add → length=$LEN"; FAILED=1; fi
echo "[g1p-smoke] verify storaged has the persistence file:"
FILE_KEY="_vectors/$NAME.lhv1"
FILE_HTTP="$(curl -sS -o /dev/null -w '%{http_code}' "http://127.0.0.1:3211/storage/get/$FILE_KEY")"
if [ "$FILE_HTTP" = "200" ]; then
echo " ✓ $FILE_KEY present in storaged"
else
echo " ✗ file=$FILE_HTTP"; FAILED=1
fi
echo "[g1p-smoke] search pre-restart:"
python3 - <<EOF > "$TMP/q.json"
import json
i = 1
v = [0.0] * $DIM
v[i % $DIM] = 1.0
v[(i+1) % $DIM] = (i % 13) * 0.01
print(json.dumps({"vector": v, "k": 1}))
EOF
TOP_PRE="$(curl -sS -X POST "http://127.0.0.1:3110/v1/vectors/index/$NAME/search" \
-H 'Content-Type: application/json' -d @"$TMP/q.json" | jq -r '.results[0].id')"
if [ "$TOP_PRE" = "w-001" ]; then echo " ✓ pre-restart top hit = w-001"; else echo " ✗ top=$TOP_PRE"; FAILED=1; fi
echo "[g1p-smoke] kill + restart vectord (rehydrate path):"
kill $VECTORD_PID 2>/dev/null || true
wait $VECTORD_PID 2>/dev/null || true
sleep 0.3
launch_vectord
sleep 0.2 # let initial Refresh complete (no API to query rehydration done)
echo "[g1p-smoke] vectord rehydrated index list shows $NAME:"
NAMES_HTTP="$(curl -sS http://127.0.0.1:3110/v1/vectors/index | jq -r '.names | length')"
if [ "$NAMES_HTTP" = "1" ]; then echo " ✓ list count=1 after restart"; else echo " ✗ count=$NAMES_HTTP"; FAILED=1; fi
INFO_LEN="$(curl -sS "http://127.0.0.1:3110/v1/vectors/index/$NAME" | jq -r '.length')"
if [ "$INFO_LEN" = "50" ]; then echo " ✓ length=50 after restart (state survived)"; else echo " ✗ length=$INFO_LEN"; FAILED=1; fi
echo "[g1p-smoke] search post-restart:"
TOP_POST="$(curl -sS -X POST "http://127.0.0.1:3110/v1/vectors/index/$NAME/search" \
-H 'Content-Type: application/json' -d @"$TMP/q.json" | jq -r '.results[0].id')"
DIST_POST="$(curl -sS -X POST "http://127.0.0.1:3110/v1/vectors/index/$NAME/search" \
-H 'Content-Type: application/json' -d @"$TMP/q.json" | jq -r '.results[0].distance')"
if [ "$TOP_POST" = "w-001" ]; then
echo " ✓ post-restart top hit = w-001 (dist=$DIST_POST)"
else
echo " ✗ post-restart top=$TOP_POST"; FAILED=1
fi
echo "[g1p-smoke] DELETE then restart → index gone:"
curl -sS -o /dev/null -X DELETE "http://127.0.0.1:3110/v1/vectors/index/$NAME"
FILE_HTTP="$(curl -sS -o /dev/null -w '%{http_code}' "http://127.0.0.1:3211/storage/get/$FILE_KEY")"
if [ "$FILE_HTTP" = "404" ]; then
echo " ✓ persistence file removed from storaged"
else
echo " ✗ file=$FILE_HTTP after delete"; FAILED=1
fi
kill $VECTORD_PID 2>/dev/null || true
wait $VECTORD_PID 2>/dev/null || true
sleep 0.3
launch_vectord
sleep 0.2
NAMES_HTTP="$(curl -sS http://127.0.0.1:3110/v1/vectors/index | jq -r '.names | length')"
if [ "$NAMES_HTTP" = "0" ]; then echo " ✓ post-delete restart list count=0"; else echo " ✗ count=$NAMES_HTTP"; FAILED=1; fi
if [ "$FAILED" -eq 0 ]; then
echo "[g1p-smoke] G1P acceptance gate: PASSED"
exit 0
else
echo "[g1p-smoke] G1P acceptance gate: FAILED"
exit 1
fi