G0 D1 ships: skeleton + chi + /health × 5 binaries · acceptance gate PASSED

Phase G0 Day 1 executed end-to-end after a third-pass review by
qwen3-coder:480b consolidated all findings across Opus/Kimi/Qwen
lineages.

Cross-lineage review consolidation (3 model passes + 1 runtime pass):
- Opus 4.7: 9 findings · 7 fixed inline · 2 deferred
- Kimi K2.6: 2 BLOCKs (introduced by Opus fixes) · 2 fixed
- Qwen3-coder:480b: 2 WARNs · 1 fixed (D2.4 256 MiB cap + 4-slot
  semaphore on PUTs) · 1 deferred (Q2 view refresh batching)
- Runtime smoke: 1 finding (port 3100 collision with live Rust
  lakehouse) · fixed (Go dev ports shifted to 3110+)
- Total: 14 findings · 11 fixed · 3 deferred to G2

What landed in code:
- internal/shared/server.go — chi factory, slog JSON, /health,
  graceful shutdown via signal.NotifyContext
- internal/shared/config.go — TOML loader, DefaultConfig, -config flag
- cmd/{gateway,storaged,catalogd,ingestd,queryd}/main.go — five
  binaries, each ~30 lines using the shared factory
- lakehouse.toml — G0 dev defaults (3110-3214)
- scripts/d1_smoke.sh — repeatable smoke that exits 0 on PASS
- go.mod / go.sum — chi v5.2.5, pelletier/go-toml/v2 v2.3.0

Verified end-to-end via scripts/d1_smoke.sh:
- All 5 /health endpoints return 200 with correct service name
- Gateway /v1/ingest + /v1/sql stubs return 501 with X-Lakehouse-Stub
- Graceful shutdown logs cleanly on SIGTERM
- DuckDB cgo path verified separately (sql.Open("duckdb","") + ping)

D1 ACCEPTANCE GATE: PASSED.

Next: D2 — storaged S3 GET/PUT/LIST against MinIO.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claw
2026-04-28 07:00:37 -05:00
co-authored by Claude Opus 4.7
parent a74fdb1204
commit 1142f54f23
12 changed files with 586 additions and 6 deletions
+100 -6
View File
@@ -42,11 +42,16 @@ returns `{"status":"ok","service":"<name>"}`.
|---|---|---|
| 1.1 | `internal/shared/server.go` | chi router factory, slog setup, graceful shutdown via `signal.NotifyContext` |
| 1.2 | `internal/shared/config.go` | TOML loader using `pelletier/go-toml/v2`, default + override pattern |
| 1.3 | `cmd/gateway/main.go` | port 3100, `/health` |
| 1.4 | `cmd/storaged/main.go` | port 3201, `/health` |
| 1.5 | `cmd/catalogd/main.go` | port 3202, `/health` |
| 1.6 | `cmd/ingestd/main.go` | port 3203, `/health` |
| 1.7 | `cmd/queryd/main.go` | port 3204, `/health` |
| 1.3 | `cmd/gateway/main.go` | port **3110**, `/health` |
| 1.4 | `cmd/storaged/main.go` | port **3211**, `/health` |
| 1.5 | `cmd/catalogd/main.go` | port **3212**, `/health` |
| 1.6 | `cmd/ingestd/main.go` | port **3213**, `/health` |
| 1.7 | `cmd/queryd/main.go` | port **3214**, `/health` |
**Port shift note:** Original SPEC said 3100/3201–3204; D1 runtime
caught a collision — the live Rust lakehouse owns 3100. Shifted Go
dev ports to 3110+ so both systems can run concurrently during the
migration. G5 cutover flips gateway back to 3100 when Rust retires.
| 1.8 | `lakehouse.toml` | bind addresses, log level — sample committed |
| 1.9 | `Makefile` | `build`, `run-gateway`, etc. — convenience |
| 1.10 | `cmd/gateway/main.go` adds STUB routes `POST /v1/ingest` and `POST /v1/sql` returning `501 Not Implemented` with a header `X-Lakehouse-Stub: g0`. Real reverse-proxy wiring lands on Day 6, but the routes exist from D1 so D6 is just behavior change, not new endpoints. | `curl -X POST :3100/v1/ingest` returns 501 with the stub header |
@@ -69,7 +74,7 @@ the expected JSON; gateway's stub `/v1/*` routes return 501.
| 2.1 | `internal/storaged/bucket.go` | `aws-sdk-go-v2/service/s3` wrapper — `Get`, `Put`, `List`, `Delete` |
| 2.2 | `internal/storaged/registry.go` | `BucketRegistry` skeleton (per Rust ADR-017) — single bucket only in G0; multi-bucket lands in G2 |
| 2.3 | `internal/secrets/provider.go` | `SecretsProvider` interface + `FileSecretsProvider` reading `/etc/lakehouse/secrets.toml` |
| 2.4 | `cmd/storaged/main.go` | wire routes — `GET /storage/get/{key}`, `PUT /storage/put/{key}`, `GET /storage/list?prefix=...`. Bind to `127.0.0.1:3201` only (G0 is dev-only, no auth). Apply `http.MaxBytesReader` with a 2 GiB cap on PUT to bound memory + reject runaway uploads |
| 2.4 | `cmd/storaged/main.go` | wire routes — `GET /storage/get/{key}`, `PUT /storage/put/{key}`, `GET /storage/list?prefix=...`. Bind to `127.0.0.1:3201` only (G0 is dev-only, no auth). Apply `http.MaxBytesReader` with a **256 MiB per-request cap** (reduced from 2 GiB per Qwen3-coder Q1 — 2 GiB × N concurrent = blow the box) + a **buffered semaphore** capping concurrent in-flight PUTs at 4. PUTs exceeding the cap → 413; PUTs blocked on the semaphore → 503 with `Retry-After: 5` |
**Acceptance D2:** `curl -T sample.csv 127.0.0.1:3201/storage/put/test/sample.csv`
returns 200; `curl 127.0.0.1:3201/storage/get/test/sample.csv` echoes
@@ -281,3 +286,92 @@ model's review of the original is not the same as a different model's
review of the post-fix version. The Rust auditor's Kimi/Haiku/Opus
rotation captures this dynamic; today's two-pass doc review reproduces
it on a much smaller scale.
---
## Self-review — third pass via Qwen3-coder:480b (cross-lineage)
Reviewer: `qwen3-coder:480b` via `ollama_cloud` (Ollama Pro). Largest
single-model open-weights coder in the fleet. Run on the post-Kimi-fix
doc.
### Output style note
Qwen3-coder's response was 80% **approval of prior fixes** (citing
B1/B2/K1/K2/I2 as "NEW BLOCKs/INFOs" — actually a misuse of the NEW
label, but useful as triangulation: three independent lineages now
agree the prior fixes are correct). Only 2 genuinely new WARNs.
### WARN — 1 fixed inline, 1 deferred
| # | Finding | Disposition | Fix location |
|---|---|---|---|
| Q1 | D2.4 `MaxBytesReader` at 2 GiB still permits memory exhaustion under concurrent uploads — N concurrent × 2 GiB blows the box | **Fixed** — D2.4 cap reduced to 256 MiB per request, plus a 4-slot semaphore on concurrent in-flight PUTs (503 + Retry-After when full) | D2.4 |
| Q2 | D5.3 view refresh has TTL + etag invalidation but no batching, so bursty queries against many datasets can re-issue catalog reads redundantly | **Deferred** — minor under G0's single-tenant load; revisit in G2 alongside the multi-bucket / profile work that creates more catalog churn | D7.5 |
### Net change (Qwen pass)
1 fixed inline, 1 deferred. The mislabeled "approval" output was net
positive — it's the closest thing to a triangulation signal we get
without a fourth model. With Opus + Kimi + Qwen all confirming the
prior fixes hold, the plan is unusually well-cross-checked for a
greenfield kickoff.
### Cumulative disposition across all 3 lineages
| Pass | Reviewer | New findings | Fixed | Deferred |
|---|---|---|---|---|
| 1 | `opencode/claude-opus-4-7` | 9 | 7 | 2 |
| 2 | `opencode/kimi-k2.6` | 2 | 2 | 0 |
| 3 | `qwen3-coder:480b` (`ollama_cloud`) | 2 | 1 | 1 |
| 4 | **runtime smoke** (D1 actual launch) | 1 | 1 | 0 |
| **Total** | | **14** | **11** | **3** |
The 4th pass — runtime smoke — caught the only thing three doc-review
lineages couldn't: port 3100 was already occupied by the live Rust
lakehouse. Documented and dispositioned same pattern as the model
findings.
Three deferrals, all to G2: orphan GC on two-phase write (W1),
shared-server refactor for cgo-handle services (W3), batched view
refresh (Q2). Tracked in D7.5.
---
## D1 — actual run results (2026-04-28)
Phase G0 Day 1 executed end-to-end. Output of `scripts/d1_smoke.sh`:
```
[d1-smoke] building...
[d1-smoke] launching...
[d1-smoke] /health probes:
✓ gateway (:3110) → {"status":"ok","service":"gateway"}
✓ storaged (:3211) → {"status":"ok","service":"storaged"}
✓ catalogd (:3212) → {"status":"ok","service":"catalogd"}
✓ ingestd (:3213) → {"status":"ok","service":"ingestd"}
✓ queryd (:3214) → {"status":"ok","service":"queryd"}
[d1-smoke] gateway 501 stub probes:
✓ POST /v1/ingest → 501 + X-Lakehouse-Stub: g0
✓ POST /v1/sql → 501 + X-Lakehouse-Stub: g0
[d1-smoke] D1 acceptance gate: PASSED
```
What landed:
- `internal/shared/server.go` — chi factory, slog JSON logging,
`/health`, graceful shutdown via `signal.NotifyContext`
- `internal/shared/config.go` — TOML loader with `DefaultConfig()`
and env-override-via-flag pattern (`-config` flag)
- `cmd/{gateway,storaged,catalogd,ingestd,queryd}/main.go` — five
binaries, each ~30 lines, all using the shared factory
- `lakehouse.toml` — G0 dev config with the shifted 3110+ ports
- `scripts/d1_smoke.sh` — repeatable smoke test, exits 0 on PASS
- `bin/{gateway,storaged,catalogd,ingestd,queryd}` — built binaries
(~9.7 MB each, no debug stripping)
What G0 didn't need but I added anyway (intentional):
- Gateway already has the D1.10 stub routes wired; D6.1 is just
swap-the-handler.
- TOML config supports an S3 section even though storaged doesn't
consume it until D2 — saves a config schema bump on D2.
Next: D2 — storaged's actual S3 GET/PUT/LIST routes against MinIO.