candidates corpus: first deep-field reality test on real staffing data

Lands the second staffing corpus and the first end-to-end reality test
through the full Go pipeline: parquet → corpusingest → embedd →
vectord → matrixd → gateway.

What's new:
  - scripts/staffing_candidates/main.go — parquet Source over
    candidates.parquet (1000 rows, 11 cols), single-chunk arrow-go
    pqarrow read. Embed text: "Candidate skills: <s>. Based in
    <city>, <state>. <years> years experience. Status: <status>.
    <first> <last>." IDs prefixed "c-" so multi-corpus merges
    against workers ("w-") stay unambiguous.
  - scripts/candidates_e2e.sh — first integration smoke that runs
    the full stack (storaged + embedd + vectord + matrixd + gateway),
    ingests via corpusingest, runs a real query through
    /v1/matrix/search, prints results. Ephemeral mode (vectord
    persistence disabled via custom toml) so re-runs don't pollute
    MinIO _vectors/ and break g1p_smoke's "only-one-persisted-index"
    assertion.

Real bug caught + fixed in corpusingest:
  When LogProgress > 0, the progress goroutine's only exit was
  ctx.Done(). With context.Background() in the production driver,
  Run hung forever after the pipeline finished. Added a stopProgress
  channel that close()s after wg.Wait(). Regression test
  TestRun_ProgressLoggerExits bounds Run's wall to 2s with
  LogProgress=50ms.

This is the bug the unit tests didn't catch because every prior test
set LogProgress: 0. Reality test surfaced it on first real-data
run — exactly the hyperfocus-and-find-architectural-weakness
property J framed as the reason for the Go pass.

End-to-end output (1000 candidates, query "Python AWS Docker
engineer in Chicago available now"):

  populate: scanned=1000 embedded=1000 added=1000 wall=3.5s
  matrix returned 5 hits in 26ms

The result quality is the interesting signal: top-5 had ZERO
Chicago candidates, ZERO active-status candidates, and the exact-
skill-match (Python,AWS,Docker) ranked #3 not #1. Pipeline works;
retrieval quality has real architectural limits (no structured
filtering, no relevance gate, semantic-only ranking dominated by
secondary signals like "1 year experience" and "engineer"). This
motivates SPEC §3.4 components 3 (relevance filter) and
eventually structured filtering — exactly the kind of finding the
deep field reality tests are supposed to surface before Enterprise
cutover.

12-smoke regression sweep all green. 9 corpusingest unit tests
including the new regression. vet clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

internal/corpusingest/ingest.go

@@ -127,11 +127,13 @@ func Run(ctx context.Context, cfg Config, src Source) (Stats, error) {
 		}()
 	}
 
+	stopProgress := make(chan struct{})
 	progressDone := make(chan struct{})
 	if cfg.LogProgress > 0 {
-		ticker := time.NewTicker(cfg.LogProgress)
 		go func() {
 			defer close(progressDone)
+			ticker := time.NewTicker(cfg.LogProgress)
+			defer ticker.Stop()
 			for {
 				select {
 				case <-ticker.C:
@@ -139,8 +141,9 @@ func Run(ctx context.Context, cfg Config, src Source) (Stats, error) {
 						"index", cfg.IndexName,
 						"embedded", atomic.LoadInt64(&totalEmbedded),
 						"added", atomic.LoadInt64(&totalAdded))
+				case <-stopProgress:
+					return
 				case <-ctx.Done():
-					ticker.Stop()
 					return
 				}
 			}
@@ -152,6 +155,7 @@ func Run(ctx context.Context, cfg Config, src Source) (Stats, error) {
 	scanned, err := drainSource(ctx, cfg, src, jobs)
 	close(jobs)
 	wg.Wait()
+	close(stopProgress) // tell the progress goroutine to exit; would otherwise hang Run forever (caught by candidates e2e 2026-04-29)
 	<-progressDone
 
 	stats := Stats{

internal/corpusingest/ingest_test.go

@@ -303,6 +303,44 @@ func TestRun_EmptyTextSkipped(t *testing.T) {
 	}
 }
 
+// TestRun_ProgressLoggerExits guards the bug caught 2026-04-29 in
+// the candidates e2e: when LogProgress > 0, the progress goroutine's
+// only exit was ctx.Done(). With context.Background() in the
+// production driver, Run hung forever after the pipeline finished.
+// This test bounds Run's wall to a few hundred ms — if it regresses,
+// the test deadline kicks in.
+func TestRun_ProgressLoggerExits(t *testing.T) {
+	fg := newFakeGateway(4)
+	srv := httptest.NewServer(fg.handler())
+	defer srv.Close()
+
+	rows := []Row{
+		{ID: "a", Text: "x", Metadata: nil},
+		{ID: "b", Text: "y", Metadata: nil},
+	}
+
+	done := make(chan error, 1)
+	go func() {
+		_, err := Run(context.Background(), Config{
+			GatewayURL:  srv.URL,
+			IndexName:   "progress_test",
+			Dimension:   4,
+			HTTPClient:  srv.Client(),
+			LogProgress: 50 * time.Millisecond,
+		}, &staticSource{rows: rows})
+		done <- err
+	}()
+
+	select {
+	case err := <-done:
+		if err != nil {
+			t.Fatalf("Run: %v", err)
+		}
+	case <-time.After(2 * time.Second):
+		t.Fatal("Run did not return within 2s — progress goroutine likely hanging")
+	}
+}
+
 func TestRun_RequiresIndexName(t *testing.T) {
 	_, err := Run(context.Background(), Config{Dimension: 4},
 		&staticSource{rows: nil})