docs: 2026-04-28 upstream survey — three SPEC-changing pivots
Pre-Phase-G0 research sweep against current Go ecosystem state. Three upstream changes that the day-of SPEC missed: 1. DuckDB Go binding ownership transferred. marcboeker/go-duckdb is deprecated as of v2.5.0 — official maintainer is now github.com/duckdb/duckdb-go/v2 (DuckDB team + Marc Boeker joint hand-off). Current v2.10502.0 / DuckDB v1.5.2. SPEC §3.1 + component table updated. 2. Official Go MCP SDK exists. Switching from mark3labs/mcp-go (community) to github.com/modelcontextprotocol/go-sdk (official, Google collaboration, v1.5.0 stable, 4.4k stars, targets MCP spec 2025-11-25). Component table updated. 3. arrow-go is on v18, not v15. v18.5.2 (March 2026) has parquet encryption fixes relevant for PII-masked safe views. PRD locked stack + SPEC component table updated. Validated unchanged: coder/hnsw (220 stars, active), chi (still the clean-architecture pick over fiber/gin/echo). Surfaced for future use: anthropics/anthropic-sdk-go (official, available for direct Claude calls bypassing opencode if ever needed), duckdb-wasm (browser-side analytics future option), IVF as HNSW fallback if recall gate fails. See docs/RESEARCH_LOG_2026-04-28.md for full survey + sources. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claw
@@ -0,0 +1,133 @@
|
||||
# Research Log — 2026-04-28
|
||||
|
||||
Survey of upstream Go ecosystem state before Phase G0 begins. Goal:
|
||||
verify the SPEC's library choices reflect the *currently maintained*
|
||||
upstream and surface anything that has shifted since the SPEC was
|
||||
drafted earlier today.
|
||||
|
||||
## Sources used
|
||||
|
||||
- Context7 — `resolve-library-id` for `go-duckdb` and `arrow-go`
|
||||
- WebSearch — six targeted queries on Go HNSW, MCP SDK, DuckDB
|
||||
bindings, Anthropic SDK, HTTP frameworks, parquet maturity
|
||||
- WebFetch — direct GitHub READMEs for `coder/hnsw`,
|
||||
`modelcontextprotocol/go-sdk`, `duckdb/duckdb-go`
|
||||
|
||||
## Findings — three SPEC-changing pivots
|
||||
|
||||
### 1. DuckDB Go binding ownership transferred
|
||||
|
||||
The original SPEC named `marcboeker/go-duckdb` as the query-engine
|
||||
library. The package has been formally **deprecated** in favor of the
|
||||
official maintainer transfer.
|
||||
|
||||
| | Before (SPEC v1) | After |
|
||||
|---|---|---|
|
||||
| Repo | `github.com/marcboeker/go-duckdb` | `github.com/duckdb/duckdb-go/v2` |
|
||||
| Maintainer | Marc Boeker (community) | DuckDB team + Marc Boeker (joint, official) |
|
||||
| Latest | (varies) | v2.10502.0 (April 2026) |
|
||||
| DuckDB engine | varies | v1.5.2 |
|
||||
| Migration | n/a | `gofmt -w -r '"github.com/marcboeker/go-duckdb/v2" -> "github.com/duckdb/duckdb-go/v2"' .` |
|
||||
|
||||
**Why this matters:** the upstream migration happened at v2.5.0; using
|
||||
the old import means we'd be on a deprecated branch from day one.
|
||||
SPEC §3.1 now names the official path.
|
||||
|
||||
### 2. Official Go MCP SDK exists (Google collaboration)
|
||||
|
||||
The original SPEC named `mark3labs/mcp-go` (community implementation).
|
||||
An official SDK now exists.
|
||||
|
||||
| | Before (SPEC v1) | After |
|
||||
|---|---|---|
|
||||
| Repo | `mark3labs/mcp-go` | `github.com/modelcontextprotocol/go-sdk` |
|
||||
| Maintainer | mark3labs (community) | MCP org + Google (official) |
|
||||
| Stars | (smaller) | 4.4k |
|
||||
| Version | n/a | v1.5.0 stable |
|
||||
| Spec compat | various | targets MCP 2025-11-25, backward-compat to 2024-11-05 |
|
||||
| OSSF Scorecard | n/a | yes |
|
||||
|
||||
**Why this matters:** the MCP server (`cmd/mcp`) is one of the most
|
||||
visible Go binaries we'll ship — it's what AI agents talk to. Using
|
||||
the official SDK aligns with the MCP spec's evolution and gets us the
|
||||
Google-tested code path.
|
||||
|
||||
### 3. arrow-go is on v18, not v15
|
||||
|
||||
The original SPEC referenced `arrow-go/v15`. Apache Arrow Go has
|
||||
shipped through **v18.5.2** (March 2026).
|
||||
|
||||
| | Before (SPEC v1) | After |
|
||||
|---|---|---|
|
||||
| Module path | `apache/arrow-go/v15` | `apache/arrow-go/v18` |
|
||||
| Latest | n/a | v18.5.2 |
|
||||
| Recent fixes | n/a | parquet decryption, large string handling, complex type read |
|
||||
|
||||
**Why this matters:** v18 has parquet encryption fixes that are
|
||||
relevant for our PII-masked safe views (per Rust ADR-017's federation
|
||||
and the production cutover work). Skipping three major versions is
|
||||
unnecessary risk.
|
||||
|
||||
## Findings — validations (SPEC unchanged)
|
||||
|
||||
### `coder/hnsw` — keep
|
||||
|
||||
220 stars, 45 commits, active CI, recent PRs/issues. Documented
|
||||
import speed of 796.85 MB/s. No deprecation signals. In-memory
|
||||
alternative to Pinecone/Weaviate, fits the no-external-vector-DB
|
||||
constraint.
|
||||
|
||||
### `chi` for HTTP routing — keep
|
||||
|
||||
Confirmed as the "clean architecture, stdlib `net/http`, zero deps"
|
||||
pick. Fiber is faster (36k req/s vs ~34k for chi/gin/echo) but uses
|
||||
fasthttp, which is off the standard library path — wrong fit for our
|
||||
"boring is good" Go ethos. Fiber stays as a documented alternative if
|
||||
hot-path performance ever proves chi insufficient.
|
||||
|
||||
### `marcboeker/go-duckdb` → `duckdb/duckdb-go/v2`
|
||||
|
||||
Already covered above as pivot #1.
|
||||
|
||||
## New things worth noting (not SPEC-changing yet)
|
||||
|
||||
### Anthropic Go SDK is official
|
||||
|
||||
`github.com/anthropics/anthropic-sdk-go` is the official Anthropic Go
|
||||
client library. We currently route Claude calls through OpenCode/Zen,
|
||||
so this isn't on the Phase G0 critical path. **Worth knowing for**:
|
||||
direct Claude API calls in `aibridge` if we ever want to bypass
|
||||
opencode (e.g., for the overseer correction loop in dev mode without
|
||||
a Zen subscription).
|
||||
|
||||
Added as a noted option in the SPEC `aibridge` row.
|
||||
|
||||
### DuckDB-Wasm exists
|
||||
|
||||
`github.com/duckdb/duckdb-wasm` brings DuckDB to the browser via
|
||||
WebAssembly with Arrow / Parquet / CSV / JSON support. **Not in scope**
|
||||
for Phase G0 but a future option if the UI ever needs client-side
|
||||
analytical queries against fetched parquet (offline analytics over a
|
||||
permit cache, etc.).
|
||||
|
||||
### IVF as an HNSW alternative
|
||||
|
||||
If during Phase G3 the HNSW recall validation gate (G3.2.C) shows
|
||||
problems, IVF (Inverted File Index) is the next-best alternative:
|
||||
faster index builds, lower memory, better filtered-search performance
|
||||
than HNSW. No first-class Go IVF library was found — would need to
|
||||
wrap FAISS via cgo or hand-roll. Documented as a fallback only.
|
||||
|
||||
## What I checked but found nothing actionable
|
||||
|
||||
- HTMX-specific Go libraries (`htmx-go`, `gotempl`, etc.) — none has
|
||||
emerged as a clear standard. Sticking with `html/template` + raw
|
||||
HTMX as the SPEC plans.
|
||||
- Langfuse Go client — OSS support varies. SPEC's "hand-roll if
|
||||
needed" stays.
|
||||
|
||||
## Outcome
|
||||
|
||||
SPEC §1 + §2 + §3.1 updated to reflect the three pivots. PRD's locked
|
||||
stack table updated to match. No phase changes, no acceptance gate
|
||||
changes, no new hard problems.
|
||||
Reference in New Issue
Block a user