profit/lakehouse
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Scheduled ingest: file watcher auto-ingests from ./inbox

Browse Source 
- Drop CSV/JSON/PDF/text into ./inbox → auto-detected → Parquet → queryable
- Polls every 10 seconds (configurable)
- Processed files moved to ./inbox/processed/
- Failed files moved to ./inbox/failed/
- Dedup: same file dropped twice = no-op
- Watcher starts automatically on gateway boot
- Tested: CSV dropped → queryable in <15s

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-03-27 20:04:40 -05:00
co-authored by Claude Opus 4.6
parent 04770c97eb
commit 294f3f6a49
6 changed files with 213 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
inbox/processed/new_candidates.csv
+4
View File
@@ -0,0 +1,4 @@
Name,Phone,Email,City,State,Skills,Years
Sarah Connor,(312) 555-0001,sarah@gmail.com,Chicago,IL,"Python,AWS,Docker",8
John Wick,(212) 555-0002,john.wick@outlook.com,New York,NY,"Java,Spring Boot,Kubernetes",12
Ellen Ripley,(713) 555-0003,ripley@yahoo.com,Houston,TX,"C++,Linux,Embedded",15
1 Name Phone Email City State Skills Years
2 Sarah Connor (312) 555-0001 sarah@gmail.com Chicago IL Python,AWS,Docker 8
3 John Wick (212) 555-0002 john.wick@outlook.com New York NY Java,Spring Boot,Kubernetes 12
4 Ellen Ripley (713) 555-0003 ripley@yahoo.com Houston TX C++,Linux,Embedded 15