Phase 6: Ingest pipeline — CSV, JSON, PDF, text file support

- ingestd crate: detect file type → parse → schema detection → Parquet → catalog
- CSV: auto-detect column types (int, float, bool, string), handles $, %, commas
  Strips dollar signs from amounts, flexible row parsing, sanitized column names
- JSON: array or newline-delimited, nested object flattening (a.b.c → a_b_c)
- PDF: text extraction via lopdf, one row per page (source_file, page_number, text)
- Text/SMS: line-based ingestion with line numbers
- Dedup: SHA-256 content hash, re-ingest same file = no-op
- Gateway: POST /ingest/file multipart upload, 256MB body limit
- Schema detection per ADR-010: ambiguous types default to String
- 12 unit tests passing (CSV parsing, JSON flattening, type inference, dedup)
- Tested: messy CSV with missing data, dollar amounts, N/A values → queryable

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-03-27 08:07:31 -05:00
co-authored by Claude Opus 4.6
parent 6740a017c7
commit bb05c4412e
25 changed files with 1261 additions and 78 deletions
+4
View File
@@ -7,6 +7,7 @@ members = [
"crates/catalogd",
"crates/queryd",
"crates/aibridge",
"crates/ingestd",
"crates/gateway",
"crates/ui",
]
@@ -38,3 +39,6 @@ opentelemetry_sdk = { version = "0.28", features = ["rt-tokio"] }
opentelemetry-stdout = { version = "0.28", features = ["trace"] }
tracing-opentelemetry = "0.29"
toml = "0.8"
csv = "1"
lopdf = "0.35"
encoding_rs = "0.8"