Phase 7: Vector index + RAG pipeline

- vectord crate: chunk → embed → store → search → RAG
- chunker: configurable chunk size + overlap, sentence-boundary aware splitting
- store: embeddings as Parquet (binary blob f32 vectors), portable format
- search: brute-force cosine similarity (works up to ~100K vectors)
- rag: full pipeline — embed question → search index → retrieve context → LLM answer
- Endpoints: POST /vectors/index, /vectors/search, /vectors/rag
- Gateway wired with vectord service
- Tested: 200 candidate resumes indexed in 5.4s, semantic search + RAG working
- 20 unit tests passing (chunker, search, ingestd, shared)
- AI gives honest "no match found" when context doesn't support an answer

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-03-27 08:12:28 -05:00
co-authored by Claude Opus 4.6
parent bb05c4412e
commit 26fc98c885
30 changed files with 771 additions and 31 deletions
Generated
+19
View File
@@ -2392,6 +2392,7 @@ dependencies = [
"tracing",
"tracing-opentelemetry",
"tracing-subscriber",
"vectord",
]
[[package]]
@@ -5407,6 +5408,24 @@ version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65"
[[package]]
name = "vectord"
version = "0.1.0"
dependencies = [
"aibridge",
"arrow",
"axum",
"bytes",
"object_store",
"parquet",
"serde",
"serde_json",
"shared",
"storaged",
"tokio",
"tracing",
]
[[package]]
name = "version_check"
version = "0.9.5"