Staffing scale test: full 500K through gateway → embedd → vectord pipeline

scripts/staffing_500k/main.go: driver that reads workers_500k.csv,
embeds combined-text per worker via /v1/embed, adds to vectord index
"workers_500k", runs canonical staffing queries against the populated
index. Reproducible end-to-end test of the staffing co-pilot pipeline
at production scale.

Run results (2026-04-29 ~02:30):
  500,000 vectors ingested in 35m 36s (~234/sec avg)
  vectord peak RSS 4.5 GB (~9 KB/vector incl. HNSW graph)
  Query latency: embed 40-59ms + search 1-3ms = ~50ms end-to-end
  GPU avg ~65% (Ollama not the bottleneck — vectord Add is)

Semantic recall on canonical queries:
  "electrician with industrial wiring": top 2 are literal Electricians (d=0.30)
  "CNC operator with first article": Assembler / Quality Techs (adjacent, d=0.24)
  "forklift driver OSHA-30": warehouse roles (d=0.33)
  "warehouse picker night shift bilingual": Material Handlers (d=0.31)
  "dental hygienist": Production Workers at d=0.49+ — correctly
    LOW-similarity, signals "no dental hygienists in this manufacturing
    dataset" rather than hallucinating a fake match.

Documented gaps:
  - storaged's 256 MiB PUT cap blocks single-file LHV1 persistence
    above ~150K vectors at d=768. Test ran with persistence disabled.
  - vectord Add is RWMutex-serialized — with GPU at 65% util this is
    the throughput cap. Concurrent Adds would be 2-3x faster but
    require careful audit of coder/hnsw thread-safety (G1 scrum
    documented two known quirks).

PHASE_G0_KICKOFF.md gains a "Staffing scale test" section with full
metrics + the gaps-surfaced list. The architectural payoff is real:
six binaries, one HTTP route, ~50ms from text query to top-K
semantically-relevant workers across 500K records.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

docs/PHASE_G0_KICKOFF.md

@@ -1316,3 +1316,102 @@ Plus shared packages: `internal/storeclient`, `internal/catalogclient`,
 g1, g1p, g2 — 9 acceptance gates, all PASS. `scripts/g1_smoke.toml`
 disables vectord persistence specifically for the in-memory API smoke
 to avoid rehydrate-from-storaged contamination.
+
+---
+
+## Staffing scale test (2026-04-29) — full 500K through the pipeline
+
+After G2 (embedd) shipped, drove the entire production-scale dataset
+through the staffing co-pilot pipeline as the architectural payoff
+test. Driver: `scripts/staffing_500k/main.go` — reads
+`workers_500k.csv`, builds combined search text per worker
+(role + city + skills + certs + resume_text), embeds via
+`/v1/embed`, adds to vectord index `workers_500k`, runs five
+canonical staffing queries.
+
+### Setup
+- Services: gateway, storaged (cap), vectord (`storaged_url=""` —
+  persistence disabled because encoded HNSW for 500K exceeds
+  storaged's 256 MiB PUT cap), embedd. catalogd/ingestd/queryd not
+  needed for this test.
+- Driver concurrency: 8 parallel embed workers, embed batch size 16,
+  add batch size effectively 16 (one batch per embed call).
+- Embedding model: `nomic-embed-text` (768-d) on RTX A4000.
+
+### Throughput at scale
+
+| Metric | Value |
+|---|---|
+| Vectors ingested | 500,000 |
+| Wall time | 35m 36s |
+| Average rate | ~234 vectors/sec end-to-end |
+| Sustained rate (steady state) | 200-215/sec, oscillating with HNSW level transitions |
+| GPU utilization | ~65% average (Ollama not the bottleneck) |
+| vectord peak RSS | 4.5 GB at 500K (~9 KB/vector incl. HNSW graph + Go runtime) |
+| Memory growth pattern | Linear ~9 MB per 1000 vectors |
+
+The bottleneck is firmly vectord's HNSW Add — log(N) with growing
+constant factors. GPU sat at ~70% during steady state, dropping to
+30-50% during HNSW level transitions. The driver's 8-way embed
+concurrency could push more if vectord's Add weren't serialized
+through one RWMutex.
+
+### Query latency at 500K
+
+All five test queries:
+- Embed: 40-59ms (Ollama doing real work on the GPU)
+- Search: **1-3ms** (HNSW with default M=16, EfSearch=20)
+- End-to-end through gateway: ~50ms total
+
+This is the architectural payoff: the actual product latency is
+embed-bound, not search-bound. Caching common-query embeddings
+would be a real win.
+
+### Semantic recall
+
+Five canonical staffing queries, top hit per query:
+
+| Query | Top result | Distance | Notes |
+|---|---|---|---|
+| `electrician with industrial wiring background` | **Electrician** at Mattoon IL | **0.30** | Top 2 are both literal Electricians |
+| `CNC operator with first article and gauge R&R experience` | Assembler at Chicago IL | 0.24 | Top 2-3 are Quality Techs (adjacent manufacturing) |
+| `forklift driver OSHA-30 certified warehouse` | Inventory Clerk at St. Louis | 0.33 | Mix of Material Handler + Warehouse Associate in top 5 |
+| `warehouse picker night shift bilingual` | Material Handler at Evansville | 0.31 | Adjacent warehouse roles |
+| `dental hygienist three years experience` | Production Worker at Madison | **0.49** | **Correctly low-similarity** — there are no dental hygienists in the manufacturing dataset; system signals "not a real match" rather than hallucinating |
+
+The "dental hygienist" result is the most architecturally honest:
+distance 0.49+ across the top-5 says "your query doesn't fit this
+dataset" instead of returning bogus matches. Small distance ≈ real
+match; big distance = no match available, judge accordingly.
+
+### What this proves
+
+1. **The full Lakehouse-Go staffing pipeline works at production
+   scale.** 500K real staffing records → 50ms similarity query
+   end-to-end through six binaries (gateway → embedd → Ollama →
+   gateway → vectord → response).
+2. **HNSW vectord scales to 500K vectors with default params.**
+   4.5 GB resident, 1-3ms search latency. Persistence (G1P) wasn't
+   used because the encoded file would exceed storaged's PUT cap;
+   that's a real G3+ cleanup but not a G1P bug.
+3. **The Provider-interface design held.** Swapping Ollama for
+   OpenAI/Voyage in G3 would change exactly one file in
+   `internal/embed/`; the rest of the stack is unaffected.
+
+### Gaps surfaced
+
+- **Persistence at scale**: storaged's 256 MiB cap blocks single-
+  file `LHV1` blobs > ~150K vectors at d=768. Either bump the cap
+  for vector workloads, split the file across multiple keys, or
+  use multipart uploads in storaged. Defer to G3+ — the test ran
+  with persistence disabled.
+- **vectord Add is single-threaded via RWMutex**. With Ollama's
+  GPU only at 65% and our embed concurrency at 8, the Add lock is
+  the throughput cap. SetMaxOpenConns(1)-style serialization made
+  sense for G1 (avoid HNSW concurrency hazards) but the staffing
+  scale test shows the cost: ~30-40 minutes per 500K dataset.
+  Concurrent Adds would be a 2-3x speedup if the library can
+  handle them; library has known thread-safety quirks (G1 scrum
+  documented two), so this needs careful audit before lifting.
+- **No caching of recent embed results**. Five queries each cost a
+  fresh ~50ms Ollama call. Common-query cache is post-G2.