profit/lakehouse
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
52 Commits 15 Branches 3 Tags
4d64e4a618cf9dd3f68ada0755e4c4f486ca366d
Commit Graph
5 Commits
Author SHA1 Message Date
rootandClaude Opus 4.6 c49b30053b Observer KB: new app for ChatGPT/Claude PDF knowledge extraction 
- New branch: observer-kb
- PRD: apps/observer-kb/docs/PRD.md
- ChatGPT PDF parser: extracts conversation turns (user/assistant)
- Upload portal: pure HTML/JS (no WASM), drag & drop, progress bars
- Browse: view ingested datasets with preview
- Search: semantic search across embedded conversations (HNSW)
- Served on :3400 via systemd (observer-kb.service)
- Uses Lakehouse core for storage, query, vectors, AI

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-27 22:23:09 -05:00
rootandClaude Opus 4.6 9992b5f135 Database connector: PostgreSQL → Parquet import 
- POST /ingest/postgres/tables — list all tables in a database
- POST /ingest/postgres/import — import table → Parquet → catalog → queryable
- Auto type mapping: int2/4/8 → Int, float4/8 → Float64, bool → Boolean,
  text/varchar/jsonb/timestamp → Utf8 (safe default per ADR-010)
- Auto PII detection + lineage on import
- Empty password support for trust auth
- Tested: imported lab_trials (40 rows, 10 cols) and threat_intel (20 rows, 30 cols)
  from local knowledge_base Postgres database — immediately queryable

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-27 20:14:16 -05:00
rootandClaude Opus 4.6 294f3f6a49 Scheduled ingest: file watcher auto-ingests from ./inbox 
- Drop CSV/JSON/PDF/text into ./inbox → auto-detected → Parquet → queryable
- Polls every 10 seconds (configurable)
- Processed files moved to ./inbox/processed/
- Failed files moved to ./inbox/failed/
- Dedup: same file dropped twice = no-op
- Watcher starts automatically on gateway boot
- Tested: CSV dropped → queryable in <15s

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-27 20:04:40 -05:00
rootandClaude Opus 4.6 35f0559d78 Phase 14: Schema evolution with AI migration rules 
- Schema diff detection: compare old vs new schema, identify changes
  (added, removed, type changed, renamed columns)
- Fuzzy rename detection: "first_name" → "full_name" detected by shared word parts
- Auto-generated migration rules: direct map, cast, concat, split, drop
  Each rule has confidence score (0.0-1.0)
- AI migration prompt builder: generates LLM prompt for complex schema changes
  LLM suggests JSON migration rules when heuristics aren't enough
- 5 new unit tests (detect added, removed, type change, rename, rule generation)
- 30 total unit tests passing

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-27 19:31:19 -05:00
rootandClaude Opus 4.6 bb05c4412e Phase 6: Ingest pipeline — CSV, JSON, PDF, text file support 
- ingestd crate: detect file type → parse → schema detection → Parquet → catalog
- CSV: auto-detect column types (int, float, bool, string), handles $, %, commas
  Strips dollar signs from amounts, flexible row parsing, sanitized column names
- JSON: array or newline-delimited, nested object flattening (a.b.c → a_b_c)
- PDF: text extraction via lopdf, one row per page (source_file, page_number, text)
- Text/SMS: line-based ingestion with line numbers
- Dedup: SHA-256 content hash, re-ingest same file = no-op
- Gateway: POST /ingest/file multipart upload, 256MB body limit
- Schema detection per ADR-010: ambiguous types default to String
- 12 unit tests passing (CSV parsing, JSON flattening, type inference, dedup)
- Tested: messy CSV with missing data, dollar amounts, N/A values → queryable

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-27 08:07:31 -05:00