Phase 6: Ingest pipeline — CSV, JSON, PDF, text file support

Browse Source

- ingestd crate: detect file type → parse → schema detection → Parquet → catalog
- CSV: auto-detect column types (int, float, bool, string), handles $, %, commas
  Strips dollar signs from amounts, flexible row parsing, sanitized column names
- JSON: array or newline-delimited, nested object flattening (a.b.c → a_b_c)
- PDF: text extraction via lopdf, one row per page (source_file, page_number, text)
- Text/SMS: line-based ingestion with line numbers
- Dedup: SHA-256 content hash, re-ingest same file = no-op
- Gateway: POST /ingest/file multipart upload, 256MB body limit
- Schema detection per ADR-010: ambiguous types default to String
- 12 unit tests passing (CSV parsing, JSON flattening, type inference, dedup)
- Tested: messy CSV with missing data, dollar amounts, N/A values → queryable

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

...

This commit is contained in:

root

2026-03-27 08:07:31 -05:00

co-authored by Claude Opus 4.6

parent 6740a017c7

commit bb05c4412e

25 changed files with 1261 additions and 78 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

data/datasets/employees.parquet

BIN

Binary file not shown.

data/datasets/events.parquet

BIN

Binary file not shown.

data/datasets/products.parquet

BIN

Binary file not shown.

data/datasets/sms_messages.parquet

BIN

Binary file not shown.

data/datasets/test_ingest.parquet

BIN

Binary file not shown.