profit/golangLAKEHOUSE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

g5 cutover: bigger load test — 5.87M req, 0 errors, 370MB RSS

Browse Source 
Larger-scale follow-up to the original load test. Three axis
expansions: corpus 200→5K workers, body variety 6→200 distinct
queries, concurrency sweep 10/50/100/200, plus mixed
embed+search workload.

Concurrency sweep on /v1/matrix/search direct (3 min each):
  conc=10:  486,733 req  · 2,704 RPS · p50 2.19ms · p99 6.7ms
  conc=50:  1,148,543 req · 6,381 RPS · p50 7.08ms · p99 20ms
  conc=100: 1,253,389 req · 6,963 RPS · p50 13.34ms · p99 37ms
  conc=200: 1,460,676 req · 8,114 RPS · p50 23.45ms · p99 56ms

Mixed embed+search at 60 conc each, 90s:
  /v1/embed: 1,127,854 req · 12,531 RPS · p50 3.31ms · p99 14.6ms
  /v1/matrix/search: 392,229 req · 4,358 RPS · p50 12.68ms · p99 33.8ms

TOTAL: 5,869,424 requests across ~13.5 minutes. ZERO errors.

Resource footprint during peak load:
  matrixd  105% CPU, 33MB RSS (bottleneck — pegs 1 core)
  vectord   39% CPU, 82MB RSS
  gateway   44% CPU, 41MB RSS
  embedd    30% CPU, 67MB RSS
  Total RSS across 11 daemons: ~370MB

Compare to Rust gateway under similar load: 14.9GB RSS, 374% CPU.
Go uses ~40x less memory + spreads load across daemons rather
than packing into one mega-process.

Saturation analysis:
- conc 10→50: +135% RPS (linear-ish scaling)
- conc 50→100: +9% RPS (saturation begins)
- conc 100→200: +17% RPS (matrixd 1-core pegged)

Headroom paths if production exceeds current demand:
1. Run multiple matrixd instances behind a load balancer.
   Substrate is stateless (recordings via storaged), horizontal
   scale is straightforward.
2. Profile matrixd's per-request work (role-gate + judge-eligibility
   + result merge).
3. Skip Bun for hot endpoints (direct nginx → Go = 5.7x previously
   measured).

Evidence: reports/cutover/g5_load_test_big.md (full tables +
methodology + repro script).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-05-01 05:18:00 -05:00
co-authored by Claude Opus 4.7
parent 2a974d6dea
commit 3a2823c02f
2 changed files with 154 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
reports/cutover/SUMMARY.md
+1
View File
@@ -15,6 +15,7 @@ what's safe to flip. Append a row when a new endpoint clears parity.
| **G5 cutover slice live** | 2026-05-01 | (none — pure cutover) | Bun `/_go/*` → Go gateway `:4110` | ✅ End-to-end | First real Bun-frontend traffic to Go substrate. Rust legacy `mcp-server/index.ts` gains opt-in `/_go/*` pass-through driven by `GO_LAKEHOUSE_URL` env (systemd drop-in at `/etc/systemd/system/lakehouse-agent.service.d/go-cutover.conf`). `/_go/v1/embed` returns nomic-embed-text-v2-moe vectors; `/_go/v1/matrix/search` returns 3/3 Forklift Operators against persistent stack's 200-worker corpus. Reversible (unset env or revert systemd unit). See `g5_first_slice_live.md`. |
| **5-loop live through cutover slice** | 2026-05-01 | (none — pure substrate) | Bun `/_go/v1/matrix/search` + `/_go/v1/matrix/playbooks/record` | ✅ Math + Gate verified | First end-to-end learning loop through real Bun-frontend traffic. Cold dist 0.4449 → warm dist 0.2224 (BoostFactor=0.5 for score=1.0; 0.4449×0.5=0.2225 expected, 0.2224 observed — 4-decimal exact). Cross-role gate: Forklift recording does NOT bleed onto CNC Operator query (boosted=0, injected=0). Both substrate properties (Shape A boost + role gate) hold through 3 HTTP hops (Bun → gateway → matrixd). See `g5_first_loop_live.md`. |
| **Production load test** | 2026-05-01 | (none — pure load probe) | Bun `/_go/v1/matrix/search` + direct Go `:4110` | ✅ 0 errors / 101k req | Three runs, **zero correctness errors**. Direct-to-Go: 2,772 RPS @ p50 2.5ms / p99 8.5ms (production-grade). Via Bun: 484 RPS @ p50 4.6ms / p99 92ms (Bun event-loop is the bottleneck — 5.7× RPS hit, 11× p99 inflation; substrate itself is fine). For staffing-domain demand (<1 RPS typical), Bun-fronted has 480× headroom. See `g5_load_test.md`. |
| **Big load test (5K corpus, 200 bodies)** | 2026-05-01 | (none — pure load probe) | Direct Go `:4110/v1/matrix/search` + `:4110/v1/embed` | ✅ **0 errors / 5.87M req** | Concurrency sweep (10/50/100/200) + mixed embed+search workload. Peak: 8,114 RPS @ conc=200 (search). Mixed: 16,889 RPS combined. Saturation at conc=100+ — matrixd pegs 1 CPU core. **Total RSS ~370MB** across 11 daemons (40× lower than Rust 14.9G). matrixd identified as horizontal-scale target. See `g5_load_test_big.md`. |
## Wire-format drift catalog