RAG reranker + manifest bucket fix — quality improvements from eval

RAG pipeline now includes a cross-encoder rerank step between retrieval
and generation. The LLM re-sorts top-K results by relevance before
they become context. Falls back to original order if model output is
unparseable (~5% with 7B models). Also improved the generation prompt
to be domain-aware ("staffing database") and request specific citations.

Fixed 4 catalog manifests with bucket="data" (pre-federation leftover)
that poisoned the entire DataFusion query context on startup. The
"users", "lab_trials", "meta_runs", and "new_candidates" datasets
now correctly reference bucket="primary". This bug was surfaced by
the quality evaluation pipeline — wouldn't have been found by
structural tests alone.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-16 22:19:11 -05:00
co-authored by Claude Opus 4.6
parent b38812481e
commit f9f92706f3
33 changed files with 2455 additions and 50 deletions
@@ -0,0 +1,26 @@
{
"id": "ef6c0acf-de7a-4be0-877e-e0c2889f390f",
"name": "threat_intel",
"schema_fingerprint": "df1e126046147b3de42086880e10c3501a3a615ecddf336bc24957a24c321241",
"objects": [
{
"bucket": "primary",
"key": "datasets/threat_intel.parquet",
"size_bytes": 247112,
"created_at": "2026-04-16T07:59:13.882669337Z"
}
],
"created_at": "2026-04-16T07:59:13.882669807Z",
"updated_at": "2026-04-16T07:59:13.882669807Z",
"description": "",
"owner": "",
"sensitivity": null,
"columns": [],
"lineage": null,
"freshness": null,
"tags": [],
"row_count": null,
"last_embedded_at": null,
"embedding_stale_since": null,
"embedding_refresh_policy": null
}