Batch 4: embed fixture-mode — partial R-006 closure
Adds cmd/fake_ollama, a minimal Ollama-API-compatible fake that
implements just enough surface for embedd to drive end-to-end
without a real Ollama install:
GET /api/tags — fixed model list including nomic-embed-text
POST /api/embeddings — deterministic dim-D vector from sha256(prompt)
GET /health — for the smoke's poll_health helper
Same prompt → bit-identical vector across runs, machines, and CI
nodes. Vectors are NOT semantically meaningful; the fake validates
the embed CONTRACT (dimension echo, response shape, status codes,
deterministic round-trip), not real semantic ranking. Real ranking
still requires real Ollama and lives in scripts/g2_smoke.sh + the
integration tier of the proof harness.
scripts/g2_smoke_fixtures.sh — full chain smoke against the fake:
- Build fake_ollama + embedd + vectord + gateway
- Start fake on :11435 (distinct from real Ollama at :11434)
- Generate temp lakehouse.toml with provider_url override
- Boot embedd/vectord/gateway with --config <override>
- 4 assertions: dim=768, deterministic same-text, different-text
divergence, bad-model → 4xx/5xx (fake 404 → embedd 502)
- Trap-cleanup tears down all 4 binaries + tmp config
Wired into the task runner:
just smoke-g2-fixtures
Closes R-006 partially:
- Embed half: ✓ — CI / fresh-clone reviewers without Ollama can
now run the embed contract smoke
- Storage half: deferred — mocking S3 protocol is non-trivial
(multipart, signed URLs, etc.) and MinIO itself is lightweight
enough to install via Docker in any CI environment. Documented
as Sprint 0 follow-up if a CI system without Docker shows up.
What this DOESN'T cover:
- Real semantic similarity (use scripts/g2_smoke.sh + real Ollama)
- Real Ollama API quirks (timeouts, version-specific shapes,
/api/embed batch endpoint that newer versions support)
Verified:
bash scripts/g2_smoke_fixtures.sh — 4/4 assertions PASS, ~3s wall
just verify — vet + test + 9 smokes still green
Doesn't replace the existing g2_smoke.sh (which still requires real
Ollama and exercises the actual embed semantics). Adds an alternate
mode for portability.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
Executable
+146
@@ -0,0 +1,146 @@
|
||||
#!/usr/bin/env bash
|
||||
# G2 smoke — fixtures variant. Same shape as g2_smoke.sh but points
|
||||
# embedd at the Go fake Ollama (cmd/fake_ollama) instead of a real
|
||||
# Ollama install. Useful for CI / fresh-clone reviewers who don't
|
||||
# have Ollama set up.
|
||||
#
|
||||
# Validates the embed contract end-to-end:
|
||||
# - POST /v1/embed → 200, dim=768
|
||||
# - Same text twice → byte-identical vector (fake is deterministic)
|
||||
# - Different texts → different vectors
|
||||
# - Bad model → 4xx (fake rejects unknown models with 404 → embedd
|
||||
# maps to 502)
|
||||
#
|
||||
# What this DOESN'T cover:
|
||||
# - Real semantic similarity (fake vectors are sha256-derived; not
|
||||
# semantically meaningful)
|
||||
# - Real Ollama API quirks (timeouts, version-specific shapes)
|
||||
#
|
||||
# Closes R-006 partial: embedd no longer needs real Ollama for the
|
||||
# CI / fresh-clone path. MinIO mocking is a separate Sprint 0
|
||||
# follow-up.
|
||||
#
|
||||
# Usage: ./scripts/g2_smoke_fixtures.sh
|
||||
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
export PATH="$PATH:/usr/local/go/bin"
|
||||
|
||||
FAKE_PORT=11435 # distinct from real Ollama at 11434
|
||||
EMBEDD_PORT=3216
|
||||
GATEWAY_PORT=3110
|
||||
VECTORD_PORT=3215
|
||||
|
||||
echo "[g2-fixtures] building fake_ollama + embedd + vectord + gateway..."
|
||||
go build -o bin/ ./cmd/fake_ollama ./cmd/embedd ./cmd/vectord ./cmd/gateway
|
||||
|
||||
pkill -f "bin/fake_ollama" 2>/dev/null || true
|
||||
pkill -f "bin/(embedd|vectord|gateway)" 2>/dev/null || true
|
||||
sleep 0.3
|
||||
|
||||
PIDS=()
|
||||
TMP="$(mktemp -d)"
|
||||
cleanup() {
|
||||
echo "[g2-fixtures] cleanup"
|
||||
for p in "${PIDS[@]}"; do [ -n "$p" ] && kill "$p" 2>/dev/null || true; done
|
||||
rm -rf "$TMP"
|
||||
}
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
poll_health() {
|
||||
local port="$1" deadline=$(($(date +%s) + 5))
|
||||
while [ "$(date +%s)" -lt "$deadline" ]; do
|
||||
if curl -sS --max-time 1 "http://127.0.0.1:$port/health" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
sleep 0.05
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# 1. Start fake_ollama on port 11435
|
||||
echo "[g2-fixtures] launching fake_ollama on :${FAKE_PORT}..."
|
||||
./bin/fake_ollama --bind "127.0.0.1:${FAKE_PORT}" --dim 768 \
|
||||
> "$TMP/fake_ollama.log" 2>&1 &
|
||||
PIDS+=($!)
|
||||
poll_health "$FAKE_PORT" || { echo "fake_ollama failed"; cat "$TMP/fake_ollama.log"; exit 1; }
|
||||
|
||||
# 2. Write override config pointing embedd at fake_ollama
|
||||
CFG="$TMP/lakehouse_fixtures.toml"
|
||||
sed "s|provider_url *= *\".*\"|provider_url = \"http://127.0.0.1:${FAKE_PORT}\"|" \
|
||||
lakehouse.toml > "$CFG"
|
||||
|
||||
# 3. Start embedd, vectord, gateway with the override config
|
||||
echo "[g2-fixtures] launching embedd/vectord/gateway with fixture config..."
|
||||
for SPEC in "vectord:${VECTORD_PORT}" "embedd:${EMBEDD_PORT}" "gateway:${GATEWAY_PORT}"; do
|
||||
NAME="${SPEC%:*}"; PORT="${SPEC#*:}"
|
||||
./bin/"$NAME" --config "$CFG" > "$TMP/${NAME}.log" 2>&1 &
|
||||
PIDS+=($!)
|
||||
if ! poll_health "$PORT"; then
|
||||
echo "[g2-fixtures] $NAME failed to bind on :$PORT"
|
||||
tail -10 "$TMP/${NAME}.log"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
# 4. Run the assertions
|
||||
FAILED=0
|
||||
|
||||
echo "[g2-fixtures] /v1/embed with one text → 200 + dim=768"
|
||||
RESP=$(curl -sS -X POST "http://127.0.0.1:${GATEWAY_PORT}/v1/embed" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"texts":["hello world"]}')
|
||||
DIM=$(echo "$RESP" | jq -r '.dimension // empty')
|
||||
N=$(echo "$RESP" | jq -r '.vectors | length')
|
||||
MODEL=$(echo "$RESP" | jq -r '.model // empty')
|
||||
if [ "$DIM" = "768" ] && [ "$N" = "1" ] && [ "$MODEL" = "nomic-embed-text" ]; then
|
||||
echo " ✓ dim=768, model=nomic-embed-text"
|
||||
else
|
||||
echo " ✗ dim=$DIM n=$N model=$MODEL"; FAILED=1
|
||||
fi
|
||||
|
||||
echo "[g2-fixtures] same text twice → byte-identical vector (deterministic)"
|
||||
V1=$(curl -sS -X POST "http://127.0.0.1:${GATEWAY_PORT}/v1/embed" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"texts":["test"]}' | jq -c '.vectors[0]')
|
||||
V2=$(curl -sS -X POST "http://127.0.0.1:${GATEWAY_PORT}/v1/embed" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"texts":["test"]}' | jq -c '.vectors[0]')
|
||||
if [ "$V1" = "$V2" ]; then
|
||||
echo " ✓ deterministic"
|
||||
else
|
||||
echo " ✗ same input → different vectors (fake should be deterministic)"; FAILED=1
|
||||
fi
|
||||
|
||||
echo "[g2-fixtures] different texts → different vectors"
|
||||
VA=$(curl -sS -X POST "http://127.0.0.1:${GATEWAY_PORT}/v1/embed" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"texts":["alpha"]}' | jq -c '.vectors[0]')
|
||||
VB=$(curl -sS -X POST "http://127.0.0.1:${GATEWAY_PORT}/v1/embed" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"texts":["beta"]}' | jq -c '.vectors[0]')
|
||||
if [ "$VA" != "$VB" ]; then
|
||||
echo " ✓ different texts diverge"
|
||||
else
|
||||
echo " ✗ different texts produced identical vectors"; FAILED=1
|
||||
fi
|
||||
|
||||
echo "[g2-fixtures] bad model → 4xx/5xx (fake returns 404, embedd maps to 502)"
|
||||
HTTP=$(curl -s -o /dev/null -w "%{http_code}" \
|
||||
-X POST "http://127.0.0.1:${GATEWAY_PORT}/v1/embed" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"texts":["x"],"model":"definitely-not-loaded"}')
|
||||
if [ "$HTTP" -ge 400 ] && [ "$HTTP" -lt 600 ]; then
|
||||
echo " ✓ unknown model → $HTTP"
|
||||
else
|
||||
echo " ✗ unknown model → $HTTP"; FAILED=1
|
||||
fi
|
||||
|
||||
if [ "$FAILED" = "0" ]; then
|
||||
echo "[g2-fixtures] ✓ G2 fixture-mode acceptance: PASSED"
|
||||
exit 0
|
||||
else
|
||||
echo "[g2-fixtures] ✗ G2 fixture-mode acceptance: FAILED"
|
||||
exit 1
|
||||
fi
|
||||
Reference in New Issue
Block a user