Background job system for embedding — fixes 100K timeout · 6a532cb248 - lakehouse

Background job system for embedding — fixes 100K timeout

- JobTracker: create/update/complete/fail jobs with progress tracking
- POST /vectors/index now returns immediately with job_id (HTTP 202)
- Embedding runs in tokio::spawn background task
- GET /vectors/jobs/{id} returns live progress (chunks embedded, rate, ETA)
- GET /vectors/jobs lists all jobs
- Progress logged every 100 batches with chunks/sec and ETA
- 100K embedding job running successfully at 44 chunks/sec
- System stays responsive during embedding (queries in 23ms)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

This commit is contained in:

root

2026-03-27 09:03:07 -05:00

co-authored by

Claude Opus 4.6

parent 354c9c4a04

commit 6a532cb248

22 changed files with 313 additions and 304 deletions

									
										crates/vectord/Cargo.toml
									
		+1
		
												View File
												
				@@ -16,3 +16,4 @@ bytes = { workspace = true }

				object_store = { workspace = true }

				parquet = { workspace = true }

				arrow = { workspace = true }

				chrono = { workspace = true }