G0 D4: ingestd CSV → Parquet → catalogd register · 2 scrum fixes

Phase G0 Day 4 ships ingestd: multipart CSV upload, Arrow schema
inference per ADR-010 (default-to-string on ambiguity), single-pass
streaming CSV → Parquet via pqarrow batched writer (Snappy compressed,
8192 rows per batch), PUT to storaged at content-addressed key
datasets/<name>/<fp_hex>.parquet, register manifest with catalogd.
Acceptance smoke 6/6 PASS including idempotent re-ingest (proves
inference is deterministic — same CSV always produces same fingerprint)
and schema-drift → 409 (proves catalogd's gate fires on ingest traffic).

Schema fingerprint is SHA-256 over (name, type) tuples in header order
using ASCII record/unit separators (0x1e/0x1f) so column names with
commas can't collide. Nullability intentionally NOT in the fingerprint
— a column gaining nulls isn't a schema change.

Cross-lineage scrum on shipped code:
  - Opus 4.7 (opencode):                       4 WARN + 3 INFO (after 2 self-retracted BLOCKs)
  - Kimi K2-0905 (openrouter):                 1 BLOCK + 2 WARN + 1 INFO
  - Qwen3-coder (openrouter):                  2 BLOCK + 2 WARN + 2 INFO

Fixed (2, both Opus single-reviewer):
  C-DRIFT: PUT-then-register on fixed datasets/<name>/data.parquet
    meant a schema-drift ingest overwrote the live parquet BEFORE
    catalogd's 409 fired → storaged inconsistent with manifest.
    Fix: content-addressed key datasets/<name>/<fp_hex>.parquet.
    Drift writes to a different file (orphan in G2 GC scope); the
    live data is never corrupted.
  C-WCLOSE: pqarrow.NewFileWriter not Closed on error paths leaks
    buffered column data + OS resources per failed ingest.
    Fix: deferred guarded close with wClosed flag.

Dismissed (5, all false positives):
  Qwen BLOCK "csv.Reader needs LazyQuotes=true for multi-line" — false,
    Go csv handles RFC 4180 multi-line quoted fields by default
  Qwen BLOCK "row[i] OOB" — already bounds-checked at schema.go:73
    and csv.go:201
  Kimi BLOCK "type assertion panic if pqarrow reorders fields" —
    speculative, no real path
  Kimi WARN + Qwen WARN×2 "RecordBuilder leak on early error" —
    false convergent. Outer defer rb.Release() captures the current
    builder; in-loop release runs before reassignment. No leak.

Deferred (6 INFO + accepted-with-rationale on 3 WARN): sample
boundary type mismatch (G0 cap bounds peak), string-match
paranoia on http.MaxBytesError, multipart double-buffer (G2 spool-
to-disk), separator validation, body close ordering, etc.

The D4 scrum produced fewer real findings than D3 (2 vs 6) — both
were architectural hazards smoke wouldn't catch because the smoke's
"schema drift → 409" assertion was passing even in the corrupted-
state world. The 409 fires correctly; what was wrong was the PUT
having already mutated the live parquet before the validation check.
Opus's PUT-then-register read of the order is exactly the kind of
architectural insight the cross-lineage scrum is designed to surface.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

internal/ingestd/catalog_client.go

@@ -0,0 +1,96 @@
+// catalog_client.go — HTTP client to catalogd. ingestd ships
+// manifests through here after writing the Parquet to storaged.
+// Symmetric in shape with internal/catalogd/store_client.go: thin
+// wrapper, drain-and-close discipline, sentinel errors for 4xx
+// classes that the handler maps back to HTTP.
+package ingestd
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"strings"
+	"time"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/catalogd"
+)
+
+// CatalogClient talks HTTP to catalogd's /catalog/* routes.
+type CatalogClient struct {
+	baseURL string
+	hc      *http.Client
+}
+
+// ErrFingerprintConflict mirrors catalogd's 409 — same name,
+// different schema fingerprint.
+var ErrFingerprintConflict = errors.New("ingestd: catalogd reports schema fingerprint conflict (409)")
+
+// NewCatalogClient builds a client against catalogd's base URL
+// (e.g. "http://127.0.0.1:3212").
+func NewCatalogClient(baseURL string) *CatalogClient {
+	return &CatalogClient{
+		baseURL: strings.TrimRight(baseURL, "/"),
+		hc:      &http.Client{Timeout: 30 * time.Second},
+	}
+}
+
+// RegisterRequest mirrors catalogd's POST /catalog/register body.
+type RegisterRequest struct {
+	Name              string             `json:"name"`
+	SchemaFingerprint string             `json:"schema_fingerprint"`
+	Objects           []catalogd.Object  `json:"objects"`
+	RowCount          *int64             `json:"row_count,omitempty"`
+}
+
+// RegisterResponse mirrors catalogd's 200/conflict response.
+type RegisterResponse struct {
+	Manifest *catalogd.Manifest `json:"manifest"`
+	Existing bool               `json:"existing"`
+}
+
+// Register POSTs to /catalog/register. Returns ErrFingerprintConflict
+// on 409, the decoded response on 200, an error on anything else.
+func (c *CatalogClient) Register(ctx context.Context, req *RegisterRequest) (*RegisterResponse, error) {
+	body, err := json.Marshal(req)
+	if err != nil {
+		return nil, fmt.Errorf("marshal register: %w", err)
+	}
+	httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/catalog/register", bytes.NewReader(body))
+	if err != nil {
+		return nil, fmt.Errorf("register req: %w", err)
+	}
+	httpReq.Header.Set("Content-Type", "application/json")
+	httpReq.ContentLength = int64(len(body))
+
+	resp, err := c.hc.Do(httpReq)
+	if err != nil {
+		return nil, fmt.Errorf("register do: %w", err)
+	}
+	defer drainAndClose(resp.Body)
+
+	if resp.StatusCode == http.StatusConflict {
+		preview, _ := io.ReadAll(io.LimitReader(resp.Body, 256))
+		return nil, fmt.Errorf("%w: %s", ErrFingerprintConflict, string(preview))
+	}
+	if resp.StatusCode != http.StatusOK {
+		preview, _ := io.ReadAll(io.LimitReader(resp.Body, 256))
+		return nil, fmt.Errorf("register status %d: %s", resp.StatusCode, string(preview))
+	}
+	var out RegisterResponse
+	if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
+		return nil, fmt.Errorf("register decode: %w", err)
+	}
+	return &out, nil
+}
+
+// drainAndClose mirrors the catalogd store_client helper — drain a
+// bounded amount of body bytes before close so HTTP/1.1 keep-alive
+// pool reuse stays healthy on error paths.
+func drainAndClose(body io.ReadCloser) {
+	_, _ = io.Copy(io.Discard, io.LimitReader(body, 64<<10))
+	_ = body.Close()
+}

internal/ingestd/csv.go

@@ -0,0 +1,232 @@
+// csv.go — single-pass CSV → Arrow → Parquet streaming pipeline.
+// Buffers the first N data rows for schema inference, then writes
+// those rows + every subsequent row directly into the pqarrow writer
+// in record batches of BatchSize. Output is the encoded Parquet
+// payload + the inferred schema + total row_count.
+package ingestd
+
+import (
+	"bytes"
+	"encoding/csv"
+	"errors"
+	"fmt"
+	"io"
+	"strconv"
+
+	"github.com/apache/arrow-go/v18/arrow"
+	"github.com/apache/arrow-go/v18/arrow/array"
+	"github.com/apache/arrow-go/v18/arrow/memory"
+	"github.com/apache/arrow-go/v18/parquet"
+	"github.com/apache/arrow-go/v18/parquet/compress"
+	"github.com/apache/arrow-go/v18/parquet/pqarrow"
+)
+
+// DefaultBatchSize is the per-record-batch row count fed to the
+// pqarrow writer. 8192 is the Arrow community default — large
+// enough to amortize per-batch overhead, small enough to keep peak
+// memory bounded for very wide CSVs.
+const DefaultBatchSize = 8192
+
+// IngestResult is what IngestCSV returns to the caller. Parquet
+// holds the full encoded payload — fits in memory by design (G0
+// CSVs are <1 GB; storaged caps PUT at 256 MiB; the gap will
+// resurface if/when we hit it).
+type IngestResult struct {
+	Schema   Schema
+	Parquet  []byte
+	RowCount int64
+}
+
+// IngestCSV reads CSV from r, infers the schema from the first
+// SampleRows data rows, and streams the entire CSV (sample + rest)
+// into a Parquet payload using the inferred schema.
+//
+// sampleRows ≤ 0 → DefaultSampleRows; batchSize ≤ 0 → DefaultBatchSize.
+//
+// The CSV is read once. The sample rows are held in memory only
+// until they're flushed to the writer; subsequent rows stream
+// through the per-column Arrow builders one batch at a time.
+func IngestCSV(r io.Reader, sampleRows, batchSize int) (*IngestResult, error) {
+	if sampleRows <= 0 {
+		sampleRows = DefaultSampleRows
+	}
+	if batchSize <= 0 {
+		batchSize = DefaultBatchSize
+	}
+
+	cr := csv.NewReader(r)
+	cr.FieldsPerRecord = -1 // tolerate ragged rows; trailing-empty cells become nulls
+
+	header, err := cr.Read()
+	if err != nil {
+		if errors.Is(err, io.EOF) {
+			return nil, errors.New("ingestd: CSV is empty (no header)")
+		}
+		return nil, fmt.Errorf("read header: %w", err)
+	}
+
+	// Buffer up to sampleRows for schema inference. Buffered rows
+	// are also written to the Parquet output — no double-read.
+	buffered := make([][]string, 0, sampleRows)
+	for len(buffered) < sampleRows {
+		row, err := cr.Read()
+		if errors.Is(err, io.EOF) {
+			break
+		}
+		if err != nil {
+			return nil, fmt.Errorf("read row %d: %w", len(buffered)+1, err)
+		}
+		buffered = append(buffered, row)
+	}
+
+	schema, err := InferSchema(header, buffered)
+	if err != nil {
+		return nil, err
+	}
+	arrowSchema := schema.ArrowSchema()
+
+	mem := memory.NewGoAllocator()
+	var buf bytes.Buffer
+	props := parquet.NewWriterProperties(
+		parquet.WithCompression(compress.Codecs.Snappy),
+	)
+	w, err := pqarrow.NewFileWriter(arrowSchema, &buf, props, pqarrow.NewArrowWriterProperties())
+	if err != nil {
+		return nil, fmt.Errorf("pqarrow writer: %w", err)
+	}
+	// Per scrum C-WCLOSE (Opus WARN): writer holds buffered column
+	// data + OS resources; every error return path past this point
+	// must close it, otherwise long-running ingestd leaks per failed
+	// ingest. closeOnce guards the success-path explicit Close from
+	// double-firing.
+	wClosed := false
+	defer func() {
+		if !wClosed {
+			_ = w.Close()
+		}
+	}()
+
+	// Build a fresh RecordBuilder for each batch — flush, release, repeat.
+	rowsTotal := int64(0)
+	flush := func(rb *array.RecordBuilder) error {
+		rec := rb.NewRecord()
+		defer rec.Release()
+		return w.Write(rec)
+	}
+
+	rb := array.NewRecordBuilder(mem, arrowSchema)
+	defer rb.Release()
+	rowsInBatch := 0
+
+	appendRow := func(row []string) error {
+		if err := appendRowToBuilders(rb, schema, row); err != nil {
+			return err
+		}
+		rowsInBatch++
+		rowsTotal++
+		if rowsInBatch >= batchSize {
+			if err := flush(rb); err != nil {
+				return err
+			}
+			rb.Release()
+			rb = array.NewRecordBuilder(mem, arrowSchema)
+			rowsInBatch = 0
+		}
+		return nil
+	}
+
+	// Replay buffered (already-read) rows into the writer first.
+	for _, row := range buffered {
+		if err := appendRow(row); err != nil {
+			return nil, fmt.Errorf("buffered row: %w", err)
+		}
+	}
+
+	// Then drain the rest of the CSV.
+	for {
+		row, err := cr.Read()
+		if errors.Is(err, io.EOF) {
+			break
+		}
+		if err != nil {
+			return nil, fmt.Errorf("read row %d: %w", rowsTotal+1, err)
+		}
+		if err := appendRow(row); err != nil {
+			return nil, err
+		}
+	}
+
+	// Flush any partial trailing batch.
+	if rowsInBatch > 0 {
+		if err := flush(rb); err != nil {
+			return nil, fmt.Errorf("flush trailing batch: %w", err)
+		}
+	}
+
+	if err := w.Close(); err != nil {
+		return nil, fmt.Errorf("pqarrow close: %w", err)
+	}
+	wClosed = true
+
+	return &IngestResult{
+		Schema:   schema,
+		Parquet:  buf.Bytes(),
+		RowCount: rowsTotal,
+	}, nil
+}
+
+// appendRowToBuilders writes one CSV row's cells into the per-
+// column builders held by rb. Cells beyond the row's width append
+// nulls (ragged-row tolerance). Empty cells append nulls on
+// numeric/bool columns and empty strings on string columns.
+func appendRowToBuilders(rb *array.RecordBuilder, schema Schema, row []string) error {
+	for i, col := range schema {
+		var cell string
+		if i < len(row) {
+			cell = row[i]
+		}
+		fb := rb.Field(i)
+		switch col.Type {
+		case TypeInt64:
+			b := fb.(*array.Int64Builder)
+			if cell == "" {
+				b.AppendNull()
+				continue
+			}
+			v, err := strconv.ParseInt(cell, 10, 64)
+			if err != nil {
+				return fmt.Errorf("col %q: %q is not int64: %w", col.Name, cell, err)
+			}
+			b.Append(v)
+		case TypeFloat64:
+			b := fb.(*array.Float64Builder)
+			if cell == "" {
+				b.AppendNull()
+				continue
+			}
+			v, err := strconv.ParseFloat(cell, 64)
+			if err != nil {
+				return fmt.Errorf("col %q: %q is not float64: %w", col.Name, cell, err)
+			}
+			b.Append(v)
+		case TypeBool:
+			b := fb.(*array.BooleanBuilder)
+			if cell == "" {
+				b.AppendNull()
+				continue
+			}
+			if !isBoolLiteral(cell) {
+				return fmt.Errorf("col %q: %q is not bool", col.Name, cell)
+			}
+			b.Append(cell == "true" || cell == "True" || cell == "TRUE")
+		default: // TypeString
+			b := fb.(*array.StringBuilder)
+			b.Append(cell)
+		}
+	}
+	return nil
+}
+
+// quiet linter — arrow.Schema is referenced via schema.ArrowSchema()
+// but the import otherwise looks unused on first scan.
+var _ = arrow.NewSchema

internal/ingestd/csv_test.go

@@ -0,0 +1,128 @@
+package ingestd
+
+import (
+	"bytes"
+	"context"
+	"strings"
+	"testing"
+
+	"github.com/apache/arrow-go/v18/arrow/array"
+	"github.com/apache/arrow-go/v18/arrow/memory"
+	"github.com/apache/arrow-go/v18/parquet/file"
+	"github.com/apache/arrow-go/v18/parquet/pqarrow"
+)
+
+func TestIngestCSV_Basic(t *testing.T) {
+	csvText := strings.Join([]string{
+		"id,name,salary,active",
+		"1,Alice,50000,true",
+		"2,Bob,60000,false",
+		"3,Carol,,true",
+		"4,Dave,75000,",
+	}, "\n")
+
+	res, err := IngestCSV(strings.NewReader(csvText), 0, 0)
+	if err != nil {
+		t.Fatalf("IngestCSV: %v", err)
+	}
+
+	if res.RowCount != 4 {
+		t.Errorf("RowCount: got %d, want 4", res.RowCount)
+	}
+	if len(res.Schema) != 4 {
+		t.Fatalf("schema cols: got %d, want 4", len(res.Schema))
+	}
+
+	want := []ColumnSpec{
+		{Name: "id", Type: TypeInt64, Nullable: false},
+		{Name: "name", Type: TypeString, Nullable: false},
+		{Name: "salary", Type: TypeInt64, Nullable: true}, // empty cell on row 3
+		{Name: "active", Type: TypeBool, Nullable: true},  // empty cell on row 4
+	}
+	for i, w := range want {
+		if res.Schema[i] != w {
+			t.Errorf("col %d: got %+v, want %+v", i, res.Schema[i], w)
+		}
+	}
+
+	// Round-trip through the pqarrow reader.
+	tbl, err := readParquetTable(res.Parquet)
+	if err != nil {
+		t.Fatalf("read parquet: %v", err)
+	}
+	defer tbl.Release()
+	if tbl.NumRows() != 4 {
+		t.Errorf("parquet rows: got %d, want 4", tbl.NumRows())
+	}
+}
+
+func TestIngestCSV_StringFallback(t *testing.T) {
+	// Per ADR-010: "salary" with mixed values → string.
+	csvText := "id,salary\n1,50000\n2,N/A\n3,60000\n"
+	res, err := IngestCSV(strings.NewReader(csvText), 0, 0)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if res.Schema[1].Type != TypeString {
+		t.Errorf("salary fell to %s, want string", res.Schema[1].Type)
+	}
+}
+
+func TestIngestCSV_BatchBoundary(t *testing.T) {
+	// 17 rows, batch size 5 → 4 batches (5+5+5+2). Tests the trailing-
+	// partial-batch flush + the schema sample being smaller than rows
+	// in the file.
+	var sb strings.Builder
+	sb.WriteString("id\n")
+	for i := 0; i < 17; i++ {
+		sb.WriteString("1\n")
+	}
+	res, err := IngestCSV(strings.NewReader(sb.String()), 5, 5)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if res.RowCount != 17 {
+		t.Errorf("RowCount: got %d, want 17", res.RowCount)
+	}
+}
+
+func TestIngestCSV_EmptyFile(t *testing.T) {
+	if _, err := IngestCSV(strings.NewReader(""), 0, 0); err == nil {
+		t.Error("empty CSV should error")
+	}
+}
+
+func TestIngestCSV_HeaderOnly(t *testing.T) {
+	res, err := IngestCSV(strings.NewReader("a,b,c\n"), 0, 0)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if res.RowCount != 0 {
+		t.Errorf("RowCount: got %d, want 0", res.RowCount)
+	}
+	// All-empty samples → all string columns per inferColumnType.
+	for _, c := range res.Schema {
+		if c.Type != TypeString {
+			t.Errorf("col %q with no samples: got %s, want string", c.Name, c.Type)
+		}
+	}
+}
+
+// readParquetTable is a small test helper.
+func readParquetTable(b []byte) (interface{ NumRows() int64; Release() }, error) {
+	rdr, err := file.NewParquetReader(bytes.NewReader(b))
+	if err != nil {
+		return nil, err
+	}
+	pr, err := pqarrow.NewFileReader(rdr, pqarrow.ArrowReadProperties{}, memory.NewGoAllocator())
+	if err != nil {
+		return nil, err
+	}
+	tbl, err := pr.ReadTable(context.Background())
+	if err != nil {
+		return nil, err
+	}
+	return tbl, nil
+}
+
+var _ = array.NewRecordBuilder // keep import for the symbol path

internal/ingestd/schema.go

@@ -0,0 +1,193 @@
+// Package ingestd holds the CSV → Arrow → Parquet → catalogd pipeline.
+// Schema inference per ADR-010: when a column is ambiguous or mixed,
+// fall back to string. Legacy CRM data is messy — a column with
+// "123", "N/A", "" is a string, not an integer. Downstream queries
+// CAST as needed.
+package ingestd
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"errors"
+	"fmt"
+	"strconv"
+	"strings"
+
+	"github.com/apache/arrow-go/v18/arrow"
+)
+
+// ColumnType is the inferred logical type of a CSV column. Maps to
+// an Arrow primitive in ArrowSchema().
+type ColumnType string
+
+const (
+	TypeString  ColumnType = "string"
+	TypeInt64   ColumnType = "int64"
+	TypeFloat64 ColumnType = "float64"
+	TypeBool    ColumnType = "bool"
+)
+
+// ColumnSpec is one column's name + inferred type + nullability.
+// Nullable is true when at least one sampled cell was empty.
+type ColumnSpec struct {
+	Name     string     `json:"name"`
+	Type     ColumnType `json:"type"`
+	Nullable bool       `json:"nullable"`
+}
+
+// Schema is the inferred CSV schema in column order. Order is
+// preserved from the CSV header — the fingerprint and Arrow schema
+// both depend on it.
+type Schema []ColumnSpec
+
+// DefaultSampleRows is how many data rows InferSchema looks at when
+// no explicit cap is supplied. 1000 is enough to catch ambiguity in
+// most real CSVs without making header inference O(file size).
+const DefaultSampleRows = 1000
+
+// InferSchema returns a Schema for the given CSV headers + sampled
+// rows. The sample is interpreted as "the first N data rows, in
+// header order"; columns shorter than the header are treated as
+// missing (nullable). Empty header names are rejected.
+func InferSchema(headers []string, samples [][]string) (Schema, error) {
+	if len(headers) == 0 {
+		return nil, errors.New("ingestd: empty CSV header")
+	}
+	for i, h := range headers {
+		if strings.TrimSpace(h) == "" {
+			return nil, fmt.Errorf("ingestd: empty header name at column %d", i)
+		}
+	}
+	cols := make(Schema, len(headers))
+	for i, name := range headers {
+		// Per-column sample slice — skip cells beyond the row's width.
+		colSamples := make([]string, 0, len(samples))
+		nullable := false
+		for _, row := range samples {
+			if i >= len(row) {
+				nullable = true
+				continue
+			}
+			cell := row[i]
+			if cell == "" {
+				nullable = true
+				continue
+			}
+			colSamples = append(colSamples, cell)
+		}
+		cols[i] = ColumnSpec{
+			Name:     name,
+			Type:     inferColumnType(colSamples),
+			Nullable: nullable,
+		}
+	}
+	return cols, nil
+}
+
+// inferColumnType walks the non-empty samples once per candidate
+// type, returning the strictest match. ADR-010: any ambiguity →
+// string. Order matters: int → float → bool → string. A column with
+// only "1"/"0" parses as int64, NOT bool — bool only when values are
+// the literal text "true"/"false".
+func inferColumnType(samples []string) ColumnType {
+	if len(samples) == 0 {
+		// All empty cells. Conservative default — string lets future
+		// inserts of any shape land without a schema-conflict 409.
+		return TypeString
+	}
+	allInt := true
+	allFloat := true
+	allBool := true
+	for _, s := range samples {
+		if allInt {
+			if _, err := strconv.ParseInt(s, 10, 64); err != nil {
+				allInt = false
+			}
+		}
+		if allFloat {
+			if _, err := strconv.ParseFloat(s, 64); err != nil {
+				allFloat = false
+			}
+		}
+		if allBool {
+			if !isBoolLiteral(s) {
+				allBool = false
+			}
+		}
+		if !allInt && !allFloat && !allBool {
+			break
+		}
+	}
+	switch {
+	case allInt:
+		return TypeInt64
+	case allFloat:
+		return TypeFloat64
+	case allBool:
+		return TypeBool
+	default:
+		return TypeString
+	}
+}
+
+// isBoolLiteral matches the strict set of strings that count as
+// boolean. Excluding "1"/"0" so they remain int64 — keeps the type
+// system honest for staffing data where 0/1 columns are typically
+// counts, not flags.
+func isBoolLiteral(s string) bool {
+	switch s {
+	case "true", "True", "TRUE", "false", "False", "FALSE":
+		return true
+	}
+	return false
+}
+
+// Fingerprint is a deterministic SHA-256 of the schema's column
+// (name, type) tuples in header order. Stable across runs; flips
+// the moment a column is added, removed, renamed, or retyped — which
+// catalogd's ADR-020 register turns into a 409 Conflict so schema
+// drift becomes loud instead of silent.
+//
+// Nullability is intentionally NOT part of the fingerprint: a column
+// that gained a null in iteration N+1 isn't "the schema changed,"
+// it's "the data got a missing value." The Arrow schema still marks
+// the column nullable via the spec's flag — the on-wire shape just
+// doesn't gate idempotency on it.
+func (s Schema) Fingerprint() string {
+	h := sha256.New()
+	for _, c := range s {
+		h.Write([]byte(c.Name))
+		h.Write([]byte{0x1e}) // ASCII record separator — can't appear in a CSV header
+		h.Write([]byte(c.Type))
+		h.Write([]byte{0x1f}) // ASCII unit separator — between columns
+	}
+	return "sha256:" + hex.EncodeToString(h.Sum(nil))
+}
+
+// ArrowSchema converts the inferred Schema to an Arrow schema for
+// use with the Parquet writer. Every field carries its Nullable
+// flag so empty cells round-trip as nulls on numeric columns.
+func (s Schema) ArrowSchema() *arrow.Schema {
+	fields := make([]arrow.Field, len(s))
+	for i, c := range s {
+		fields[i] = arrow.Field{
+			Name:     c.Name,
+			Type:     arrowTypeFor(c.Type),
+			Nullable: c.Nullable,
+		}
+	}
+	return arrow.NewSchema(fields, nil)
+}
+
+func arrowTypeFor(t ColumnType) arrow.DataType {
+	switch t {
+	case TypeInt64:
+		return arrow.PrimitiveTypes.Int64
+	case TypeFloat64:
+		return arrow.PrimitiveTypes.Float64
+	case TypeBool:
+		return arrow.FixedWidthTypes.Boolean
+	default:
+		return arrow.BinaryTypes.String
+	}
+}

internal/ingestd/schema_test.go

@@ -0,0 +1,115 @@
+package ingestd
+
+import (
+	"testing"
+)
+
+func TestInferSchema_CleanInts(t *testing.T) {
+	headers := []string{"id", "count"}
+	samples := [][]string{{"1", "100"}, {"2", "200"}, {"3", "300"}}
+	got, err := InferSchema(headers, samples)
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, c := range got {
+		if c.Type != TypeInt64 {
+			t.Errorf("%s: got %s, want int64", c.Name, c.Type)
+		}
+		if c.Nullable {
+			t.Errorf("%s should not be nullable", c.Name)
+		}
+	}
+}
+
+func TestInferSchema_FloatColumns(t *testing.T) {
+	headers := []string{"price", "weight"}
+	samples := [][]string{{"1.5", "2.0"}, {"100", "3.14"}, {"0.0", "0"}}
+	got, _ := InferSchema(headers, samples)
+	// "price" has 1.5 + "100" + 0.0 → float64 (one of the values isn't int-parseable in 1.5)
+	if got[0].Type != TypeFloat64 {
+		t.Errorf("price: got %s, want float64", got[0].Type)
+	}
+	if got[1].Type != TypeFloat64 {
+		t.Errorf("weight: got %s, want float64", got[1].Type)
+	}
+}
+
+func TestInferSchema_AmbiguousFallsToString(t *testing.T) {
+	// ADR-010: a column with "123", "N/A", and "" is a string, not int.
+	headers := []string{"salary"}
+	samples := [][]string{{"50000"}, {"N/A"}, {"60000"}, {""}}
+	got, _ := InferSchema(headers, samples)
+	if got[0].Type != TypeString {
+		t.Errorf("salary: got %s, want string (ADR-010 fallback)", got[0].Type)
+	}
+	if !got[0].Nullable {
+		t.Errorf("salary: should be nullable (saw empty cell)")
+	}
+}
+
+func TestInferSchema_BoolLiterals(t *testing.T) {
+	headers := []string{"active", "deleted"}
+	samples := [][]string{{"true", "false"}, {"True", "False"}, {"TRUE", "FALSE"}}
+	got, _ := InferSchema(headers, samples)
+	if got[0].Type != TypeBool {
+		t.Errorf("active: got %s, want bool", got[0].Type)
+	}
+	if got[1].Type != TypeBool {
+		t.Errorf("deleted: got %s, want bool", got[1].Type)
+	}
+}
+
+func TestInferSchema_OneZeroIsInt_NotBool(t *testing.T) {
+	// Keeps the type system honest — 1/0 columns in CRM data are
+	// typically counts (children, certs), not flags.
+	headers := []string{"children"}
+	samples := [][]string{{"0"}, {"1"}, {"2"}, {"0"}}
+	got, _ := InferSchema(headers, samples)
+	if got[0].Type != TypeInt64 {
+		t.Errorf("children: got %s, want int64 (1/0 is int, not bool)", got[0].Type)
+	}
+}
+
+func TestInferSchema_EmptyHeader(t *testing.T) {
+	if _, err := InferSchema(nil, nil); err == nil {
+		t.Error("nil headers should error")
+	}
+	if _, err := InferSchema([]string{"valid", ""}, nil); err == nil {
+		t.Error("empty header name should error")
+	}
+}
+
+func TestFingerprint_Deterministic(t *testing.T) {
+	s1, _ := InferSchema([]string{"id", "name"}, [][]string{{"1", "alice"}})
+	s2, _ := InferSchema([]string{"id", "name"}, [][]string{{"1", "alice"}})
+	if s1.Fingerprint() != s2.Fingerprint() {
+		t.Errorf("fingerprint not deterministic: %s vs %s", s1.Fingerprint(), s2.Fingerprint())
+	}
+}
+
+func TestFingerprint_FlipsOnTypeChange(t *testing.T) {
+	intSchema, _ := InferSchema([]string{"id"}, [][]string{{"1"}, {"2"}})
+	strSchema, _ := InferSchema([]string{"id"}, [][]string{{"1"}, {"abc"}})
+	if intSchema.Fingerprint() == strSchema.Fingerprint() {
+		t.Error("fingerprint should flip when column type changes")
+	}
+}
+
+func TestFingerprint_StableUnderNullable(t *testing.T) {
+	// Adding null cells doesn't flip the fingerprint — it's only
+	// about (name, type), not nullability.
+	a, _ := InferSchema([]string{"id"}, [][]string{{"1"}, {"2"}})
+	b, _ := InferSchema([]string{"id"}, [][]string{{"1"}, {"2"}, {""}})
+	if a.Fingerprint() != b.Fingerprint() {
+		t.Error("fingerprint shouldn't flip when nullability changes")
+	}
+}
+
+func TestFingerprint_RespectsColumnOrder(t *testing.T) {
+	// Same columns, swapped order → different fingerprint.
+	a, _ := InferSchema([]string{"id", "name"}, [][]string{{"1", "x"}})
+	b, _ := InferSchema([]string{"name", "id"}, [][]string{{"x", "1"}})
+	if a.Fingerprint() == b.Fingerprint() {
+		t.Error("fingerprint should be order-sensitive")
+	}
+}

internal/shared/config.go

@@ -22,12 +22,21 @@ type Config struct {
 	Gateway  ServiceConfig `toml:"gateway"`
 	Storaged ServiceConfig `toml:"storaged"`
 	Catalogd CatalogConfig `toml:"catalogd"`
-	Ingestd  ServiceConfig `toml:"ingestd"`
+	Ingestd  IngestConfig  `toml:"ingestd"`
 	Queryd   ServiceConfig `toml:"queryd"`
 	S3       S3Config      `toml:"s3"`
 	Log      LogConfig     `toml:"log"`
 }
 
+// IngestConfig adds ingestd-specific knobs. ingestd needs to PUT
+// parquet to storaged AND register manifests with catalogd, so it
+// holds two upstream URLs in addition to its own bind.
+type IngestConfig struct {
+	Bind        string `toml:"bind"`
+	StoragedURL string `toml:"storaged_url"`
+	CatalogdURL string `toml:"catalogd_url"`
+}
+
 // CatalogConfig adds catalogd-specific knobs on top of the standard
 // bind. StoragedURL points at the storaged service for manifest
 // persistence; G0 defaults to the localhost bind.
@@ -66,8 +75,12 @@ func DefaultConfig() Config {
 		Gateway:  ServiceConfig{Bind: "127.0.0.1:3110"},
 		Storaged: ServiceConfig{Bind: "127.0.0.1:3211"},
 		Catalogd: CatalogConfig{Bind: "127.0.0.1:3212", StoragedURL: "http://127.0.0.1:3211"},
-		Ingestd:  ServiceConfig{Bind: "127.0.0.1:3213"},
-		Queryd:   ServiceConfig{Bind: "127.0.0.1:3214"},
+		Ingestd: IngestConfig{
+			Bind:        "127.0.0.1:3213",
+			StoragedURL: "http://127.0.0.1:3211",
+			CatalogdURL: "http://127.0.0.1:3212",
+		},
+		Queryd: ServiceConfig{Bind: "127.0.0.1:3214"},
 		S3: S3Config{
 			Endpoint:     "http://localhost:9000",
 			Region:       "us-east-1",