Scheduled ingest: file watcher auto-ingests from ./inbox

Browse Source

- Drop CSV/JSON/PDF/text into ./inbox → auto-detected → Parquet → queryable
- Polls every 10 seconds (configurable)
- Processed files moved to ./inbox/processed/
- Failed files moved to ./inbox/failed/
- Dedup: same file dropped twice = no-op
- Watcher starts automatically on gateway boot
- Tested: CSV dropped → queryable in <15s

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

...

This commit is contained in:

root

2026-03-27 20:04:40 -05:00

co-authored by Claude Opus 4.6

parent 04770c97eb

commit 294f3f6a49

6 changed files with 213 additions and 0 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

data/datasets/new_candidates.parquet

BIN

Binary file not shown.