Phase 8: Hot cache + incremental delta updates
- MemCache: LRU in-memory cache for hot datasets (configurable max, default 16GB) Pin/evict/stats endpoints: POST /query/cache/pin, /cache/evict, GET /cache/stats - Delta store: append-only delta Parquet files for row-level updates Write deltas without rewriting base files, merge at query time - Compaction: POST /query/compact merges deltas into base Parquet - Query engine: checks cache first, falls back to Parquet, merges deltas - Benchmarked on 2.47M rows: 1M row JOIN: 854ms cold → 96ms hot (8.9x speedup) 100K filter: 62ms cold → 21ms hot (3x speedup) 1.1M rows cached in 408MB RAM Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -30,8 +30,9 @@ async fn main() {
|
||||
tracing::warn!("catalog rebuild failed (empty store?): {e}");
|
||||
}
|
||||
|
||||
// Query engine
|
||||
let engine = queryd::context::QueryEngine::new(registry.clone(), store.clone());
|
||||
// Query engine with 16GB memory cache (configurable)
|
||||
let cache = queryd::cache::MemCache::new(16 * 1024 * 1024 * 1024); // 16GB
|
||||
let engine = queryd::context::QueryEngine::new(registry.clone(), store.clone(), cache);
|
||||
|
||||
// AI sidecar client
|
||||
let ai_client = aibridge::client::AiClient::new(&config.sidecar.url);
|
||||
|
||||
Reference in New Issue
Block a user