Real-scale validation post-G0: configurable ingest cap + workers_500k metrics

Validated G0 substrate against the production workers_500k.parquet
dataset (18 cols × 500,000 rows). Findings + one applied fix:

Finding #1 (FIXED): ingestd's hardcoded 256 MiB cap rejected the 500K
CSV (344 MiB) with 413. Cap fired correctly, no OOM. Extracted to
[ingestd].max_ingest_bytes config field; default 256 MiB, override
per deployment for known-large workloads. With cap bumped to 512 MiB,
500K ingest succeeds in 3.12s with ingestd peak RSS 209 MiB.

Finding #2 (deferred): ingestd doesn't release memory between
ingests. Go runtime conservative; long-running daemon, fine.

Finding #3: DuckDB-via-httpfs is healthy at 500K. GROUP BY 45ms,
count(*) 24ms, AVG 47ms, schema introspection 25ms. Sub-linear
scaling vs 100K — the s3:// read path is not a bottleneck.

Finding #4: ADR-010 type inference correctly handled real staffing
data. worker_id → BIGINT, numeric scores → DOUBLE, multi-line
resume_text → VARCHAR. 1000-row sample sufficient.

Finding #5: Go's encoding/csv handles RFC 4180 quoted-comma fields
and multi-line quoted text without LazyQuotes — confirming the D4
scrum's dismissal of Qwen's BLOCK on this point.

Net: substrate handles production-scale data with one config knob.
No correctness issues, no OOMs, no silent type errors.
All 6 G0 smokes still PASS after the cap-config change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

docs/PHASE_G0_KICKOFF.md

@@ -1173,3 +1173,103 @@ Real architectural choices that proved out:
 Next: G1+. The substrate is now in place to build gRPC adapters,
 vector indices (Lance/HNSW), the Go MCP SDK port, distillation
 rebuild, and observer/Langfuse integration on top.
+
+---
+
+## Real-scale validation (2026-04-29 post-G0)
+
+After G0 shipped, a half-day validation against the real production
+dataset (`workers_500k.parquet` from the Rust system, 18 cols ×
+500,000 rows of staffing data with quoted-comma fields, multi-line
+text, mixed numeric/categorical). The smokes use 5-row CSVs; this
+exercises the substrate at production scale.
+
+### Method
+
+Converted `/home/profit/lakehouse/data/datasets/workers_500k.parquet`
+→ CSV via DuckDB (`COPY (SELECT * FROM read_parquet) TO 'csv'`).
+Drove the resulting CSV through the gateway's `/v1/ingest` route,
+queried via `/v1/sql`. Captured wall time, parquet output size,
+and ingestd peak RSS.
+
+### Results
+
+| Dataset | CSV size | Parquet | Ingest wall | ingestd peak RSS |
+|---|---|---|---|---|
+| 100K × 18 cols | 68 MiB | 14 MiB | 0.93s | 98 MiB |
+| 500K × 18 cols | 344 MiB | 71 MiB | 3.12s | 209 MiB |
+
+| Query (500K rows) | Latency |
+|---|---|
+| `SELECT count(*)` | 24ms |
+| `SELECT col, name FROM x ORDER BY id LIMIT 5` | 56ms |
+| `SELECT state, count(*) GROUP BY state ORDER BY n DESC LIMIT 10` | 45ms |
+| `SELECT round(avg(reliability), 4), min, max` | 47ms |
+| `SELECT count(*) WHERE state='CA' AND reliability > 0.8` | 34ms |
+| `SELECT column_name, data_type FROM duckdb_columns()` | 25ms |
+
+Memory at rest after ingest:
+- storaged: 55 MiB
+- catalogd: 28 MiB
+- ingestd: 209 MiB (held the post-ingest heap; not released)
+- gateway: 10 MiB
+- queryd: 112 MiB
+
+### Findings
+
+**Finding #1 — ingestd's hardcoded cap is the binding constraint
+for big CSVs.** A 500K-row staffing CSV is 344 MiB. The previous
+hardcoded `maxIngestBytes = 256 << 20` rejected it with 413 in
+360ms — the cap fired correctly, no OOM, server stayed alive, but
+the dataset couldn't ship. **Fixed**: extracted to
+`[ingestd].max_ingest_bytes` config field with a 256 MiB default.
+Operators can bump for known-large workloads (validated at 512 MiB
+for the 500K dataset). The cap is on the multipart upload body,
+not on the resulting Parquet (which compresses ~5× via Snappy and
+is well under storaged's 256 MiB PUT cap).
+
+**Finding #2 — ingestd doesn't release memory between ingests.**
+Peak RSS stayed at 209 MiB after the 500K ingest finished, even
+though the CSV bytes + Arrow builders were eligible for GC. Go's
+runtime is conservative about returning heap memory to the OS.
+For a long-running daemon this is fine (next ingest reuses the
+heap); for a short-lived ingest CLI tool it would matter.
+**Deferred** — operational note, not a correctness bug.
+
+**Finding #3 — DuckDB-via-httpfs latency is healthy at 500K.**
+GROUP BY across 500K rows in 45ms. Sub-linear scaling vs 100K
+(36ms → 45ms for 5× rows). The `read_parquet('s3://...')` path
+through MinIO is not a bottleneck at this scale.
+
+**Finding #4 — Real-data type inference works.** ADR-010's
+default-to-string-on-ambiguity correctly typed `worker_id` as
+BIGINT, numeric scores as DOUBLE, multi-line `resume_text` as
+VARCHAR. No silent type errors. The 1000-row sample was sufficient.
+
+**Finding #5 — Multipart parsing handles complex CSVs.** The real
+data has quoted-comma fields (skills: `"bilingual, cold storage,
+hazmat"`) and multi-line text inside quotes (`resume_text` with
+embedded newlines). Go's `encoding/csv` defaults handle RFC 4180
+correctly without `LazyQuotes` — confirming the D4 scrum dismissal
+of Qwen's BLOCK on this point.
+
+### Decisions
+
+- **Configurable `max_ingest_bytes`**: applied. Default 256 MiB,
+  override via `[ingestd].max_ingest_bytes` in TOML.
+- **Streaming-spool multipart parser**: deferred. The current
+  in-memory ParseMultipartForm with the 64 MiB threshold spills
+  larger bodies to /tmp automatically, which is good enough for
+  production workloads up to ~1 GiB CSV. True streaming Parquet
+  generation (read CSV row-by-row from the multipart stream
+  without ever holding the full body) is a G2+ refinement.
+- **Memory release**: deferred. `runtime/debug.FreeOSMemory()`
+  after each ingest would aggressively release; not worth the
+  complexity for a long-running daemon.
+
+### Net assessment
+
+G0's substrate handles real production-scale data with one config
+knob bumped. No correctness issues, no OOMs, no silent type
+errors. Query latency is fast enough for ad-hoc analytics.
+The substrate is ready for G1+ work to build on top.