Phase 8: Hot cache + incremental delta updates

- MemCache: LRU in-memory cache for hot datasets (configurable max, default 16GB)
  Pin/evict/stats endpoints: POST /query/cache/pin, /cache/evict, GET /cache/stats
- Delta store: append-only delta Parquet files for row-level updates
  Write deltas without rewriting base files, merge at query time
- Compaction: POST /query/compact merges deltas into base Parquet
- Query engine: checks cache first, falls back to Parquet, merges deltas
- Benchmarked on 2.47M rows:
  1M row JOIN: 854ms cold → 96ms hot (8.9x speedup)
  100K filter: 62ms cold → 21ms hot (3x speedup)
  1.1M rows cached in 408MB RAM

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-03-27 08:37:28 -05:00
co-authored by Claude Opus 4.6
parent eae51977ab
commit 6df904a03c
23 changed files with 518 additions and 131 deletions
@@ -0,0 +1,15 @@
{
"id": "dce14141-f679-481b-9b48-13438cbfe057",
"name": "email_log",
"schema_fingerprint": "auto",
"objects": [
{
"bucket": "data",
"key": "datasets/email_log.parquet",
"size_bytes": 16768671,
"created_at": "2026-03-27T13:36:52.383853471Z"
}
],
"created_at": "2026-03-27T13:36:52.383859356Z",
"updated_at": "2026-03-27T13:36:52.383859356Z"
}