raw-corpus dump + vectorization + chicago contract inference pipeline
Three new pieces, executed in order:
scripts/dump_raw_corpus.sh
- One-shot bash that creates MinIO bucket `raw` and uploads all
testing corpora as a persistent immutable test set. 365 MB total
across 5 prefixes (chicago, entities, sec, staffing, llm_team)
+ MANIFEST.json. Sources: workers_500k.parquet (309 MB),
resumes.parquet, entities.jsonl, sec_company_tickers.json,
Chicago permits last 30d (2,853 records, 5.4 MB), 9 LLM Team
Postgres tables dumped via row_to_json.
scripts/vectorize_raw_corpus.ts
- Bun script that fetches each raw-bucket source via mc, runs a
source-specific extractor into {id, text} docs, posts to
/vectors/index, polls job to completion. Verified results:
chicago_permits_v1: 3,420 chunks
entity_brief_v1: 634 chunks
sec_tickers_v1: 10,341 chunks (after extractor fix for
wrapped {rows: {...}} JSON shape)
llm_team_runs_v1: in flight, 19K+ chunks
llm_team_response_cache_v1: queued
scripts/analyze_chicago_contracts.ts
- Real inference pipeline that picks N high-cost permits with
named contractors from the raw bucket, queries all 6 contract-
analysis corpora in parallel via /vectors/search, builds a
MATRIX CONTEXT preamble, calls Grok 4.1 fast for structured
staffing analysis, hand-reviews each via observer /review,
appends to data/_kb/contract_analyses.jsonl.
tests/real-world/scrum_master_pipeline.ts
- MATRIX_CORPORA_FOR_TASK extended with two new task classes:
contract_analysis (chicago + entity_brief + sec + llm_team_runs
+ llm_team_response_cache + distilled_procedural)
staffing_inference (workers_500k_v8 + entity_brief + chicago
+ llm_team_runs + distilled_procedural)
scrum_review unchanged.
This is the first time the matrix architecture operates on real
ingested data instead of code-review smoke tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -448,12 +448,35 @@ async function fetchProvenApproaches(
|
||||
// to indexes that actually contain code-review-relevant context — staffing
|
||||
// data (workers_500k_*, resumes_*) is excluded by design.
|
||||
const MATRIX_CORPORA_FOR_TASK: Record<string, string[]> = {
|
||||
// Code review task — distilled facts/procedures/hints from prior reviews
|
||||
// plus team-run history. Staffing data deliberately excluded.
|
||||
scrum_review: [
|
||||
"distilled_factual_v20260423095819",
|
||||
"distilled_procedural_v20260423102847",
|
||||
"distilled_config_hint_v20260423102847",
|
||||
"kb_team_runs_v1",
|
||||
],
|
||||
// Chicago contract / permit analysis — pulls actual permits + contractor
|
||||
// entities + SEC tickers + LLM team historical reasoning + lake-house
|
||||
// distilled procedures (which encode prior task patterns).
|
||||
contract_analysis: [
|
||||
"chicago_permits_v1",
|
||||
"entity_brief_v1",
|
||||
"sec_tickers_v1",
|
||||
"llm_team_runs_v1",
|
||||
"llm_team_response_cache_v1",
|
||||
"distilled_procedural_v20260423102847",
|
||||
],
|
||||
// Staffing inference — workers data + entity briefs + Chicago permit
|
||||
// demand signal + LLM team patterns. workers_500k_v8 is the most
|
||||
// recent dense version.
|
||||
staffing_inference: [
|
||||
"workers_500k_v8",
|
||||
"entity_brief_v1",
|
||||
"chicago_permits_v1",
|
||||
"llm_team_runs_v1",
|
||||
"distilled_procedural_v20260423102847",
|
||||
],
|
||||
};
|
||||
|
||||
interface MatrixHit {
|
||||
|
||||
Reference in New Issue
Block a user