Phase 8: Hot cache + incremental delta updates

- MemCache: LRU in-memory cache for hot datasets (configurable max, default 16GB)
  Pin/evict/stats endpoints: POST /query/cache/pin, /cache/evict, GET /cache/stats
- Delta store: append-only delta Parquet files for row-level updates
  Write deltas without rewriting base files, merge at query time
- Compaction: POST /query/compact merges deltas into base Parquet
- Query engine: checks cache first, falls back to Parquet, merges deltas
- Benchmarked on 2.47M rows:
  1M row JOIN: 854ms cold → 96ms hot (8.9x speedup)
  100K filter: 62ms cold → 21ms hot (3x speedup)
  1.1M rows cached in 408MB RAM

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

This commit is contained in:

root

2026-03-27 08:37:28 -05:00

co-authored by

Claude Opus 4.6

parent eae51977ab

commit 6df904a03c

23 changed files with 518 additions and 131 deletions

Cargo.lock

Generated

View File

@@ -3963,6 +3963,7 @@ dependencies = [
  "axum",
  "bytes",
  "catalogd",
  "chrono",
  "datafusion",
  "futures",
  "object_store",