Scheduled ingest: file watcher auto-ingests from ./inbox

- Drop CSV/JSON/PDF/text into ./inbox → auto-detected → Parquet → queryable
- Polls every 10 seconds (configurable)
- Processed files moved to ./inbox/processed/
- Failed files moved to ./inbox/failed/
- Dedup: same file dropped twice = no-op
- Watcher starts automatically on gateway boot
- Tested: CSV dropped → queryable in <15s

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-03-27 20:04:40 -05:00
co-authored by Claude Opus 4.6
parent 04770c97eb
commit 294f3f6a49
6 changed files with 213 additions and 0 deletions
+7
View File
@@ -108,6 +108,13 @@ async fn main() {
.allow_headers(Any))
.layer(TraceLayer::new_for_http());
// File watcher — auto-ingest from ./inbox
let _watcher = ingestd::watcher::spawn_watcher(
ingestd::watcher::WatchConfig::default(),
store.clone(),
registry.clone(),
);
// Start gRPC server on port+1
let grpc_port = config.gateway.port + 1;
let catalog_grpc = catalogd::grpc::CatalogGrpc::new(registry);