profit/golangLAKEHOUSE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

materializer + replay ports + vectord substrate fix verified at scale

Browse Source 
Two threads landing together — the doc edits interleave so they ship
in a single commit.

1. **vectord substrate fix verified at original scale** (closes the
   2026-05-01 thread). Re-ran multitier 5min @ conc=50: 132,211
   scenarios at 438/sec, 6/6 classes at 0% failure (was 4/6 pre-fix).
   Throughput dropped 1,115 → 438/sec because previously-broken
   scenarios now do real HNSW Add work — honest cost of correctness.
   The fix (i.vectors side-store + safeGraphAdd recover wrappers +
   smallIndexRebuildThreshold=32 + saveTask coalescing) holds at the
   footprint that originally surfaced the bug.

2. **Materializer port** — internal/materializer + cmd/materializer +
   scripts/materializer_smoke.sh. Ports scripts/distillation/transforms.ts
   (12 transforms) + build_evidence_index.ts (idempotency, day-partition,
   receipt). On-wire JSON shape matches TS so Bun and Go runs are
   interchangeable. 14 tests green.

3. **Replay port** — internal/replay + cmd/replay +
   scripts/replay_smoke.sh. Ports scripts/distillation/replay.ts
   (retrieve → bundle → /v1/chat → validate → log). Closes audit-FULL
   phase 7 live invocation on the Go side. Both runtimes append to the
   same data/_kb/replay_runs.jsonl (schema=replay_run.v1). 14 tests green.

Side effect on internal/distillation/types.go: EvidenceRecord gained
prompt_tokens, completion_tokens, and metadata fields to mirror the TS
shape the materializer transforms produce.

STATE_OF_PLAY refreshed to 2026-05-02; ARCHITECTURE_COMPARISON decisions
tracker moves the materializer + replay items from _open_ to DONE and
adds the substrate-fix scale verification row.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-05-02 03:31:02 -05:00
co-authored by Claude Opus 4.7
parent 277884b5eb
commit 89ca72d471
26 changed files with 3961 additions and 124 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/distillation/types.go
+11 -3
View File
@@ -182,9 +182,17 @@ type EvidenceRecord struct {
HumanOverride *HumanOverride `json:"human_override,omitempty"`
CostUSD float64 `json:"cost_usd,omitempty"`
LatencyMs int64 `json:"latency_ms,omitempty"`
Text string `json:"text,omitempty"`
CostUSD float64 `json:"cost_usd,omitempty"`
LatencyMs int64 `json:"latency_ms,omitempty"`
PromptTokens int64 `json:"prompt_tokens,omitempty"`
CompletionTokens int64 `json:"completion_tokens,omitempty"`
Text string `json:"text,omitempty"`
// Domain-specific bucket for source-row fields that don't earn a
// top-level slot. e.g. contract_analyses carries `contractor` here.
// Typed scalar values only — keep this small or it becomes a junk
// drawer. Mirrors EvidenceRecord.metadata in evidence_record.ts.
Metadata map[string]any `json:"metadata,omitempty"`
}
// RetrievedContext captures what the model saw via retrieval. Matches