raw-corpus dump + vectorization + chicago contract inference pipeline

Three new pieces, executed in order:

scripts/dump_raw_corpus.sh
- One-shot bash that creates MinIO bucket `raw` and uploads all
  testing corpora as a persistent immutable test set. 365 MB total
  across 5 prefixes (chicago, entities, sec, staffing, llm_team)
  + MANIFEST.json. Sources: workers_500k.parquet (309 MB),
  resumes.parquet, entities.jsonl, sec_company_tickers.json,
  Chicago permits last 30d (2,853 records, 5.4 MB), 9 LLM Team
  Postgres tables dumped via row_to_json.

scripts/vectorize_raw_corpus.ts
- Bun script that fetches each raw-bucket source via mc, runs a
  source-specific extractor into {id, text} docs, posts to
  /vectors/index, polls job to completion. Verified results:
    chicago_permits_v1: 3,420 chunks
    entity_brief_v1:    634 chunks
    sec_tickers_v1:    10,341 chunks (after extractor fix for
                        wrapped {rows: {...}} JSON shape)
    llm_team_runs_v1:  in flight, 19K+ chunks
    llm_team_response_cache_v1: queued

scripts/analyze_chicago_contracts.ts
- Real inference pipeline that picks N high-cost permits with
  named contractors from the raw bucket, queries all 6 contract-
  analysis corpora in parallel via /vectors/search, builds a
  MATRIX CONTEXT preamble, calls Grok 4.1 fast for structured
  staffing analysis, hand-reviews each via observer /review,
  appends to data/_kb/contract_analyses.jsonl.

tests/real-world/scrum_master_pipeline.ts
- MATRIX_CORPORA_FOR_TASK extended with two new task classes:
  contract_analysis (chicago + entity_brief + sec + llm_team_runs
    + llm_team_response_cache + distilled_procedural)
  staffing_inference (workers_500k_v8 + entity_brief + chicago
    + llm_team_runs + distilled_procedural)
  scrum_review unchanged.

This is the first time the matrix architecture operates on real
ingested data instead of code-review smoke tests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-25 18:44:27 -05:00
co-authored by Claude Opus 4.7
parent a496ced848
commit ed83754f20
4 changed files with 623 additions and 0 deletions
+23
View File
@@ -448,12 +448,35 @@ async function fetchProvenApproaches(
// to indexes that actually contain code-review-relevant context — staffing
// data (workers_500k_*, resumes_*) is excluded by design.
const MATRIX_CORPORA_FOR_TASK: Record<string, string[]> = {
// Code review task — distilled facts/procedures/hints from prior reviews
// plus team-run history. Staffing data deliberately excluded.
scrum_review: [
"distilled_factual_v20260423095819",
"distilled_procedural_v20260423102847",
"distilled_config_hint_v20260423102847",
"kb_team_runs_v1",
],
// Chicago contract / permit analysis — pulls actual permits + contractor
// entities + SEC tickers + LLM team historical reasoning + lake-house
// distilled procedures (which encode prior task patterns).
contract_analysis: [
"chicago_permits_v1",
"entity_brief_v1",
"sec_tickers_v1",
"llm_team_runs_v1",
"llm_team_response_cache_v1",
"distilled_procedural_v20260423102847",
],
// Staffing inference — workers data + entity briefs + Chicago permit
// demand signal + LLM team patterns. workers_500k_v8 is the most
// recent dense version.
staffing_inference: [
"workers_500k_v8",
"entity_brief_v1",
"chicago_permits_v1",
"llm_team_runs_v1",
"distilled_procedural_v20260423102847",
],
};
interface MatrixHit {