G0 D1 ships: skeleton + chi + /health × 5 binaries · acceptance gate PASSED
Phase G0 Day 1 executed end-to-end after a third-pass review by
qwen3-coder:480b consolidated all findings across Opus/Kimi/Qwen
lineages.
Cross-lineage review consolidation (3 model passes + 1 runtime pass):
- Opus 4.7: 9 findings · 7 fixed inline · 2 deferred
- Kimi K2.6: 2 BLOCKs (introduced by Opus fixes) · 2 fixed
- Qwen3-coder:480b: 2 WARNs · 1 fixed (D2.4 256 MiB cap + 4-slot
semaphore on PUTs) · 1 deferred (Q2 view refresh batching)
- Runtime smoke: 1 finding (port 3100 collision with live Rust
lakehouse) · fixed (Go dev ports shifted to 3110+)
- Total: 14 findings · 11 fixed · 3 deferred to G2
What landed in code:
- internal/shared/server.go — chi factory, slog JSON, /health,
graceful shutdown via signal.NotifyContext
- internal/shared/config.go — TOML loader, DefaultConfig, -config flag
- cmd/{gateway,storaged,catalogd,ingestd,queryd}/main.go — five
binaries, each ~30 lines using the shared factory
- lakehouse.toml — G0 dev defaults (3110-3214)
- scripts/d1_smoke.sh — repeatable smoke that exits 0 on PASS
- go.mod / go.sum — chi v5.2.5, pelletier/go-toml/v2 v2.3.0
Verified end-to-end via scripts/d1_smoke.sh:
- All 5 /health endpoints return 200 with correct service name
- Gateway /v1/ingest + /v1/sql stubs return 501 with X-Lakehouse-Stub
- Graceful shutdown logs cleanly on SIGTERM
- DuckDB cgo path verified separately (sql.Open("duckdb","") + ping)
D1 ACCEPTANCE GATE: PASSED.
Next: D2 — storaged S3 GET/PUT/LIST against MinIO.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
+100
-6
@@ -42,11 +42,16 @@ returns `{"status":"ok","service":"<name>"}`.
|
||||
|---|---|---|
|
||||
| 1.1 | `internal/shared/server.go` | chi router factory, slog setup, graceful shutdown via `signal.NotifyContext` |
|
||||
| 1.2 | `internal/shared/config.go` | TOML loader using `pelletier/go-toml/v2`, default + override pattern |
|
||||
| 1.3 | `cmd/gateway/main.go` | port 3100, `/health` |
|
||||
| 1.4 | `cmd/storaged/main.go` | port 3201, `/health` |
|
||||
| 1.5 | `cmd/catalogd/main.go` | port 3202, `/health` |
|
||||
| 1.6 | `cmd/ingestd/main.go` | port 3203, `/health` |
|
||||
| 1.7 | `cmd/queryd/main.go` | port 3204, `/health` |
|
||||
| 1.3 | `cmd/gateway/main.go` | port **3110**, `/health` |
|
||||
| 1.4 | `cmd/storaged/main.go` | port **3211**, `/health` |
|
||||
| 1.5 | `cmd/catalogd/main.go` | port **3212**, `/health` |
|
||||
| 1.6 | `cmd/ingestd/main.go` | port **3213**, `/health` |
|
||||
| 1.7 | `cmd/queryd/main.go` | port **3214**, `/health` |
|
||||
|
||||
**Port shift note:** Original SPEC said 3100/3201–3204; D1 runtime
|
||||
caught a collision — the live Rust lakehouse owns 3100. Shifted Go
|
||||
dev ports to 3110+ so both systems can run concurrently during the
|
||||
migration. G5 cutover flips gateway back to 3100 when Rust retires.
|
||||
| 1.8 | `lakehouse.toml` | bind addresses, log level — sample committed |
|
||||
| 1.9 | `Makefile` | `build`, `run-gateway`, etc. — convenience |
|
||||
| 1.10 | `cmd/gateway/main.go` adds STUB routes `POST /v1/ingest` and `POST /v1/sql` returning `501 Not Implemented` with a header `X-Lakehouse-Stub: g0`. Real reverse-proxy wiring lands on Day 6, but the routes exist from D1 so D6 is just behavior change, not new endpoints. | `curl -X POST :3100/v1/ingest` returns 501 with the stub header |
|
||||
@@ -69,7 +74,7 @@ the expected JSON; gateway's stub `/v1/*` routes return 501.
|
||||
| 2.1 | `internal/storaged/bucket.go` | `aws-sdk-go-v2/service/s3` wrapper — `Get`, `Put`, `List`, `Delete` |
|
||||
| 2.2 | `internal/storaged/registry.go` | `BucketRegistry` skeleton (per Rust ADR-017) — single bucket only in G0; multi-bucket lands in G2 |
|
||||
| 2.3 | `internal/secrets/provider.go` | `SecretsProvider` interface + `FileSecretsProvider` reading `/etc/lakehouse/secrets.toml` |
|
||||
| 2.4 | `cmd/storaged/main.go` | wire routes — `GET /storage/get/{key}`, `PUT /storage/put/{key}`, `GET /storage/list?prefix=...`. Bind to `127.0.0.1:3201` only (G0 is dev-only, no auth). Apply `http.MaxBytesReader` with a 2 GiB cap on PUT to bound memory + reject runaway uploads |
|
||||
| 2.4 | `cmd/storaged/main.go` | wire routes — `GET /storage/get/{key}`, `PUT /storage/put/{key}`, `GET /storage/list?prefix=...`. Bind to `127.0.0.1:3201` only (G0 is dev-only, no auth). Apply `http.MaxBytesReader` with a **256 MiB per-request cap** (reduced from 2 GiB per Qwen3-coder Q1 — 2 GiB × N concurrent = blow the box) + a **buffered semaphore** capping concurrent in-flight PUTs at 4. PUTs exceeding the cap → 413; PUTs blocked on the semaphore → 503 with `Retry-After: 5` |
|
||||
|
||||
**Acceptance D2:** `curl -T sample.csv 127.0.0.1:3201/storage/put/test/sample.csv`
|
||||
returns 200; `curl 127.0.0.1:3201/storage/get/test/sample.csv` echoes
|
||||
@@ -281,3 +286,92 @@ model's review of the original is not the same as a different model's
|
||||
review of the post-fix version. The Rust auditor's Kimi/Haiku/Opus
|
||||
rotation captures this dynamic; today's two-pass doc review reproduces
|
||||
it on a much smaller scale.
|
||||
|
||||
---
|
||||
|
||||
## Self-review — third pass via Qwen3-coder:480b (cross-lineage)
|
||||
|
||||
Reviewer: `qwen3-coder:480b` via `ollama_cloud` (Ollama Pro). Largest
|
||||
single-model open-weights coder in the fleet. Run on the post-Kimi-fix
|
||||
doc.
|
||||
|
||||
### Output style note
|
||||
Qwen3-coder's response was 80% **approval of prior fixes** (citing
|
||||
B1/B2/K1/K2/I2 as "NEW BLOCKs/INFOs" — actually a misuse of the NEW
|
||||
label, but useful as triangulation: three independent lineages now
|
||||
agree the prior fixes are correct). Only 2 genuinely new WARNs.
|
||||
|
||||
### WARN — 1 fixed inline, 1 deferred
|
||||
|
||||
| # | Finding | Disposition | Fix location |
|
||||
|---|---|---|---|
|
||||
| Q1 | D2.4 `MaxBytesReader` at 2 GiB still permits memory exhaustion under concurrent uploads — N concurrent × 2 GiB blows the box | **Fixed** — D2.4 cap reduced to 256 MiB per request, plus a 4-slot semaphore on concurrent in-flight PUTs (503 + Retry-After when full) | D2.4 |
|
||||
| Q2 | D5.3 view refresh has TTL + etag invalidation but no batching, so bursty queries against many datasets can re-issue catalog reads redundantly | **Deferred** — minor under G0's single-tenant load; revisit in G2 alongside the multi-bucket / profile work that creates more catalog churn | D7.5 |
|
||||
|
||||
### Net change (Qwen pass)
|
||||
|
||||
1 fixed inline, 1 deferred. The mislabeled "approval" output was net
|
||||
positive — it's the closest thing to a triangulation signal we get
|
||||
without a fourth model. With Opus + Kimi + Qwen all confirming the
|
||||
prior fixes hold, the plan is unusually well-cross-checked for a
|
||||
greenfield kickoff.
|
||||
|
||||
### Cumulative disposition across all 3 lineages
|
||||
|
||||
| Pass | Reviewer | New findings | Fixed | Deferred |
|
||||
|---|---|---|---|---|
|
||||
| 1 | `opencode/claude-opus-4-7` | 9 | 7 | 2 |
|
||||
| 2 | `opencode/kimi-k2.6` | 2 | 2 | 0 |
|
||||
| 3 | `qwen3-coder:480b` (`ollama_cloud`) | 2 | 1 | 1 |
|
||||
| 4 | **runtime smoke** (D1 actual launch) | 1 | 1 | 0 |
|
||||
| **Total** | | **14** | **11** | **3** |
|
||||
|
||||
The 4th pass — runtime smoke — caught the only thing three doc-review
|
||||
lineages couldn't: port 3100 was already occupied by the live Rust
|
||||
lakehouse. Documented and dispositioned same pattern as the model
|
||||
findings.
|
||||
|
||||
Three deferrals, all to G2: orphan GC on two-phase write (W1),
|
||||
shared-server refactor for cgo-handle services (W3), batched view
|
||||
refresh (Q2). Tracked in D7.5.
|
||||
|
||||
---
|
||||
|
||||
## D1 — actual run results (2026-04-28)
|
||||
|
||||
Phase G0 Day 1 executed end-to-end. Output of `scripts/d1_smoke.sh`:
|
||||
|
||||
```
|
||||
[d1-smoke] building...
|
||||
[d1-smoke] launching...
|
||||
[d1-smoke] /health probes:
|
||||
✓ gateway (:3110) → {"status":"ok","service":"gateway"}
|
||||
✓ storaged (:3211) → {"status":"ok","service":"storaged"}
|
||||
✓ catalogd (:3212) → {"status":"ok","service":"catalogd"}
|
||||
✓ ingestd (:3213) → {"status":"ok","service":"ingestd"}
|
||||
✓ queryd (:3214) → {"status":"ok","service":"queryd"}
|
||||
[d1-smoke] gateway 501 stub probes:
|
||||
✓ POST /v1/ingest → 501 + X-Lakehouse-Stub: g0
|
||||
✓ POST /v1/sql → 501 + X-Lakehouse-Stub: g0
|
||||
[d1-smoke] D1 acceptance gate: PASSED
|
||||
```
|
||||
|
||||
What landed:
|
||||
- `internal/shared/server.go` — chi factory, slog JSON logging,
|
||||
`/health`, graceful shutdown via `signal.NotifyContext`
|
||||
- `internal/shared/config.go` — TOML loader with `DefaultConfig()`
|
||||
and env-override-via-flag pattern (`-config` flag)
|
||||
- `cmd/{gateway,storaged,catalogd,ingestd,queryd}/main.go` — five
|
||||
binaries, each ~30 lines, all using the shared factory
|
||||
- `lakehouse.toml` — G0 dev config with the shifted 3110+ ports
|
||||
- `scripts/d1_smoke.sh` — repeatable smoke test, exits 0 on PASS
|
||||
- `bin/{gateway,storaged,catalogd,ingestd,queryd}` — built binaries
|
||||
(~9.7 MB each, no debug stripping)
|
||||
|
||||
What G0 didn't need but I added anyway (intentional):
|
||||
- Gateway already has the D1.10 stub routes wired; D6.1 is just
|
||||
swap-the-handler.
|
||||
- TOML config supports an S3 section even though storaged doesn't
|
||||
consume it until D2 — saves a config schema bump on D2.
|
||||
|
||||
Next: D2 — storaged's actual S3 GET/PUT/LIST routes against MinIO.
|
||||
|
||||
Reference in New Issue
Block a user