profit/lakehouse
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Compare commits

base: profit/lakehouse:25b7e6c3a782c11b22216baa7833628394d6383f
Branches Tags
profit/lakehouse:main profit/lakehouse:demo/post-pr11-polish-2026-04-28 profit/lakehouse:infra/replace-gpt-oss-2026-04-28 profit/lakehouse:scrum/auto-apply-19814 profit/lakehouse:scrum/iter5-prep-2026-04-24 profit/lakehouse:test/enrich-prd-pipeline profit/lakehouse:cohesion/python-inventory-and-integration profit/lakehouse:phase/45-slice-4 profit/lakehouse:phase/45-slice-3 profit/lakehouse:ops/auditor-systemd-units profit/lakehouse:fix/upsert-outcome-update-merge profit/lakehouse:auditor/scaffold profit/lakehouse:fix/upsert-outcome-serde profit/lakehouse:federation-hnsw-trials profit/lakehouse:observer-kb
profit/lakehouse:demo-latest profit/lakehouse:distillation-v1.0.0 profit/lakehouse:v0.26-memory-complete
...
compare: profit/lakehouse:auditor/scaffold
Branches Tags
profit/lakehouse:demo/post-pr11-polish-2026-04-28 profit/lakehouse:main profit/lakehouse:infra/replace-gpt-oss-2026-04-28 profit/lakehouse:scrum/auto-apply-19814 profit/lakehouse:scrum/iter5-prep-2026-04-24 profit/lakehouse:test/enrich-prd-pipeline profit/lakehouse:cohesion/python-inventory-and-integration profit/lakehouse:phase/45-slice-4 profit/lakehouse:phase/45-slice-3 profit/lakehouse:ops/auditor-systemd-units profit/lakehouse:fix/upsert-outcome-update-merge profit/lakehouse:auditor/scaffold profit/lakehouse:fix/upsert-outcome-serde profit/lakehouse:federation-hnsw-trials profit/lakehouse:observer-kb
profit/lakehouse:demo-latest profit/lakehouse:distillation-v1.0.0 profit/lakehouse:v0.26-memory-complete
58 Commits
25b7e6c3a7 ... auditor/scaffold
Author SHA1 Message Date
profit c33c1bcbc5 Auditor: poller + live end-to-end proof
lakehouse/auditor all checks passed (4 findings, all info) 
auditor/index.ts (task #9) — the top-level poller. 90s interval,
dedupes by head SHA via data/_auditor/state.json, supports --once
for CLI testing. Env gates: LH_AUDITOR_RUN_DYNAMIC=1 to include
the hybrid fixture (default off; it mutates live state),
LH_AUDITOR_SKIP_INFERENCE=1 for fast runs without cloud calls.

Single-shot run proof (task #10):

  cycle 1: 2 open PRs
    audit PR #2 f0a3ed68 "Fix: UpsertOutcome newtype serde panic"
       verdict=block, 9 findings (1 block, 5 warn, 3 info)
    audit PR #1 039ed324 "Auditor: PR-claim hard-block reviewer"
       verdict=approve, 4 findings (0 block, 0 warn, 4 info)
    audits_run=2, state persisted

Commit statuses and issue comments posted live to Gitea. PR #2 is
currently hard-blocked (lakehouse/auditor commit status = failure);
PR #1 has a passing status. State survives restart — next cycle
skips already-audited SHAs.

Both PRs now have the audit comment with per-check breakdown.
Operator can read the comment, fix blocking findings (or defend
them with a reply), push a new commit; auditor re-audits on new
SHA, verdict updates, merge gate responds accordingly.

The full loop J asked for is closed:
  1. static check caught own Phase 45 placeholder (b933334)
  2. hybrid fixture caught UpsertOutcome serde panic (9c893fb)
  3. LLM-Team-style codereview caught ternary bug (5bbcaf4)
  4. auditor poller now runs on every open PR, block/approve with
     evidence, re-audits on new SHAs

Tasks done: 1-11 (except 12, a scoped follow-up fix for UPDATE
branch dropping doc_refs). The auditor is running, catching real
bugs in its own build, and gating merges.
 2026-04-22 04:02:36 -05:00
profit 039ed32411 Auditor: KB query check + verdict orchestrator + Gitea poster
lakehouse/auditor all checks passed (4 findings, all info) 
auditor/checks/kb_query.ts (task #7) — reads data/_kb/outcomes.jsonl,
error_corrections.jsonl, data/_observer/ops.jsonl, data/_bot/cycles/*.
Cheap/offline: no model calls, tail-reads only. Fail-rate >30% in
recent scenario outcomes → warn; otherwise info. Live-proven: 1
finding emitted against current KB state (69 scenario runs, 27.7%
fail rate — below warn threshold).

auditor/audit.ts (task #8) — orchestrator. Runs static + dynamic +
inference + kb_query in parallel, calls assembleVerdict, persists
to data/_auditor/verdicts/, posts to Gitea (commit status + issue
comment). AuditOptions supports skip_dynamic/skip_inference/dry_run
for iteration.

auditor/gitea.ts — added postIssueComment (author can comment on
own PR, unlike postReview which self-review-blocks).

static.ts — skip BLOCK_PATTERNS scan on auditor/checks/* and
auditor/fixtures/* because those files legitimately contain the
patterns as regex/string-literal data. WARN/INFO patterns (TODO
comments, hardcoded placeholders) still run. Live-proven: dry-run
audit of PR #1 after fix went from 13 block findings to 0 from
static; 11 warn from inference still fire on real overreach claims.

Dry-run audit against PR #1, skip_dynamic=true:
  verdict: block (BEFORE the static fix)
  verdict: request_changes (AFTER — inference correctly flagged
           "tasks 1-9 complete" as not backed; 0 false-positive
           blocks from static self-match)
  42.5s total across checks (mostly cloud inference: 36s)
  26 claims, 39KB diff

Tasks 5 + 6 + 7 + 8 complete. Remaining: #9 (poller) + #10
(end-to-end proof) + #12 (upsert UPDATE merge fix).
 2026-04-22 03:59:38 -05:00
profit efc7b5ac44 Auditor: dynamic + inference checks 
auditor/checks/dynamic.ts — wraps runHybridFixture, maps layer
results to Findings. Placeholder-style errors (404/unimplemented/
slice N) → info; other failures → warn. Always emits a summary
finding with real numbers (shipped/placeholder phase counts + per-
layer latency). Live-tested against current stack: 2 info findings,
0 warnings — all shipped layers actually work.

auditor/checks/inference.ts — wraps the run_codereview reviewer
pattern from llm_team_ui.py, adapted for claim-vs-diff verification.
Calls /v1/chat provider=ollama_cloud model=gpt-oss:120b. Requests
strict JSON response with claim_verdicts[] and unflagged_gaps[]. A
strong claim marked "not backed" by cloud → BLOCK severity; moderate
→ warn; weak → info. Cloud-unreachable or unparseable-output → info
(never blocks on the reviewer being down).

Live-tested against PR #1 (this PR, 20 claims, 39KB diff):
  - 36.9s round-trip
  - 7 block + 23 warn + 2 info findings
  - gpt-oss:120b correctly flagged "Fully-functional auditor (tasks
    1-9 complete)" as not-backed (only 6/10 tasks done at that
    commit) — accurate catch
  - Some false positives from the original 15KB truncation threshold
    (cloud missed gitea.ts, flagged "no Gitea client present")
  - Bumped MAX_DIFF_CHARS from 15000 to 40000 to fit the full PR
    diff in context; reviewer precision improves accordingly

Tasks 5 + 6 completed. Remaining: #7 (KB query), #8 (verdict +
Gitea poster), #9 (poller), #10 (end-to-end proof), #12 (upsert
UPDATE-drops-doc_refs).
 2026-04-22 03:54:18 -05:00
profit c5da680add Fixture: unique-per-run nonce eliminates state-pollution false positive 
After the serde fix (PR #2, fix/upsert-outcome-serde) landed on main,
re-running this fixture STILL reported "doc_refs field is empty" —
but with a different root cause than the panic.

Root cause: pre-fix runs panicked on response serialization but had
already added entries to state (panic happened between upsert_entry
returning and the handler's serde_json::json! of the response). So
state.json was polluted with __auditor_test_worker__ entries from
those runs, WITHOUT doc_refs (doc_refs wasn't even wired at the time
those state rows were written).

The fixture's `find(endorsed_names.includes(TEST_WORKER_NAME))` was
picking the oldest polluted entry, not the fresh one.

Compounding: discovered a secondary bug while investigating —
upsert_entry's UPDATE branch only merges endorsed_names. doc_refs,
schema_fingerprint, valid_until on an UPDATE are silently dropped.
Filed as task #12, separate PR to follow.

Fix in this fixture: use a nonce suffix on both TEST_WORKER_NAME and
TEST_OPERATION so every run is guaranteed to hit the ADD path in
upsert_entry, sidestepping the UPDATE bug AND eliminating state
pollution entirely.

Live re-run after this edit:
  ✓ Phase 38    /v1/chat            449ms, 42 tokens
  ✓ Phase 40    Langfuse trace       20ms
  ✓ Phase 45.1  seed + doc_refs     239ms, doc_refs.length=1 persisted
  ✓ Phase 45.2  bridge diff           2ms, drifted=true
  ✗ Phase 45.3  drift-check           HONEST 404 (endpoint not built)

shipped_phases: [38, 40, 45.1, 45.2]  (was [38, 40, 45.2])
placeholder:    [45.3]                 (was [45.1, 45.3])

One fewer placeholder — exactly because the serde fix merged on
fix/upsert-outcome-serde and the fixture now cleanly exercises the
path. The loop is:
  fixture finds bug → PR fixes bug → fixture re-run confirms fix →
  one fewer placeholder.
 2026-04-22 03:50:46 -05:00
profit 5bbcaf4c33 Fix: layer-2 Langfuse filter used meaningless ternary 
Caught by running a side-test through LLM Team's run_codereview
flow (gpt-oss:120b reviewer) against this fixture, 2026-04-22.

BEFORE:
  const ourStart = Date.parse(
    l1.evidence.match(/tokens=/) ? result.ran_at : result.ran_at
  );
  // Both branches return result.ran_at — the ternary is meaningless.
  // result.ran_at is the fixture start time, NOT the moment we fired
  // /v1/chat. Any trace created between fixture-start and chat-fetch
  // would false-negative.

AFTER:
  const chat_request_sent_ms = Date.now();  // captured before layer 1
  // ...
  const recent = items.filter(t =>
    Date.parse(t.timestamp) >= chat_request_sent_ms
  );

Re-ran the fixture against the live stack — layers 1,2,4 still pass
(no regression); layer 2 trace matched at age=2494ms which is within
the chat-to-trace propagation window. Layers 3,5 still fail for the
original unrelated reasons (UpsertOutcome serde panic + Phase 45
slice 3 endpoint not built).

First concrete act-on-finding from a code-checker run. The process
works.
 2026-04-22 03:44:36 -05:00
profit 9c893fbb8c Auditor: hybrid fixture — found a pre-existing bug on first live run 
auditor/fixtures/hybrid_38_40_45.ts — the never-before-run hybrid
test. Exercises Phase 38 /v1/chat → Phase 40 Langfuse → Phase 45
slice 1 seed+doc_refs → Phase 45 slice 2 bridge drift → (expected-
fail) Phase 45 slice 3 drift-check endpoint.

auditor/fixtures/cli.ts — standalone runner. Human-readable summary
to stderr, machine-readable JSON to stdout, exit code 0/1/2 for
pass / fail / partial_pass.

Live run results — honest measurements, not hand-waved:
  ✓ Phase 38     /v1/chat returns 9 visible tokens, 6.7s latency
                 ("docker run is a common Docker command.")
  ✓ Phase 40     Langfuse trace 18a8a0b7 landed in 2.5s
  ✗ Phase 45.1   seed endpoint returns empty reply — discovered a
                 PRE-EXISTING BUG unrelated to doc_refs:

                 playbook_memory.rs:257 UpsertOutcome has newtype
                 variants Added(String) and Noop(String) under
                 #[serde(tag="mode")] — serde panics on serialize.

                 panicked at crates/vectord/src/service.rs:2323:
                 Error("cannot serialize tagged newtype variant
                 UpsertOutcome::Added containing a string")

                 Reproduced: curl /seed with AND without doc_refs
                 both get "Empty reply from server" (socket closed
                 mid-response). This bug has existed since Phase 26
                 shipped (commit 640db8c, 2026-04-21). No test or
                 caller in the repo exercised the response path live
                 against the gateway until this fixture did.

  ✓ Phase 45.2   context7 bridge confirms drift: current hash
                 475a0396ca436bba vs our stale input, upstream last
                 updated 2026-04-20
  ✗ Phase 45.3   /doc_drift/check endpoint — correctly unreachable
                 because layer 3 blocked us from getting a playbook_id;
                 endpoint still doesn't exist independent of that

Real numbers published: per-layer latency_ms, token counts,
trace_age_ms, library_id, current_hash_length. All stored in the
JSON output for downstream audit.

Value delivered: the fixture's first live run found a bug that
unit tests, compile checks, and my own "phase shipped" commits all
missed. Exactly the gap J called out — the auditor is doing what
it's supposed to do.

Bug fix is a SEPARATE concern: new task #11 tracks a separate PR
(fix/upsert-outcome-serde) so the audit finding and the fix stay
cleanly attributed.
 2026-04-22 03:34:20 -05:00
profit b933334ae2 Auditor: static diff check — catches own Phase 45 placeholder 
auditor/checks/static.ts — grep-style scan of PR diffs, no AST,
no LLM. High-signal patterns only.

Severity grading:
- BLOCK — unimplemented!(), todo!(), panic!("not implemented"),
  throw new Error("not implemented")
- WARN  — TODO/FIXME/XXX/HACK in added lines;
          new pub struct fields with <2 mentions in the diff
          (added but nobody reads it — placeholder state)
- INFO  — hardcoded "placeholder"/"dummy"/"foobar"/"changeme"/"xxx"
          strings in added lines

Live-proven — the existential test J asked for:

  vs PR #1 (scaffold):        0 findings (all scaffold fields cross-
                              reference within the diff)
  vs commit 2a4b81b (Phase    5 WARN: every DocRef field (tool,
  45 first slice — I          version_seen, snippet_hash, source_url,
  half-admitted placeholder): seen_at) added with 0 read-sites in
                              the diff

That's the auditor flagging my own "Phase 45 first slice" commit as
state-without-consumer, which is exactly what I half-admitted it
was. If PR #1 had required auditor-pass (branch protection), the
DocRef commit would have been blocked pre-merge. The auditor works
because it agreed with the honest read.

Next: dynamic hybrid test fixture (task #4) — the never-run multi-
layer pipeline test.
 2026-04-22 03:29:31 -05:00
profit bfe8985233 Auditor: claim parser 
auditor/claim_parser.ts — reads PR body + commit messages, extracts
ship-claims. Regex-based, intentionally not LLM-driven: the parser's
job is to surface claim substrates, not to judge them (that's the
inference check's job, runs later with cloud model).

Three strength tiers:
- strong   — "verified end-to-end", "live-proven", "production-ready",
             "phase N shipped", "proven"
- moderate — "shipped", "landed", "green", "passing", "works",
             "complete", "done"
- weak     — "should work", "expected to", "probably"

Live-proven against PR #1 (this PR): 4 claims extracted from
1 commit (2 strong, 2 moderate). "live-proven" correctly tagged as
strong (it IS a stronger claim than "shipped").

Next: static diff check consumes these claims + the PR diff to find
placeholder patterns — empty fns, TODO, unwired fields, etc.
 2026-04-22 03:28:06 -05:00
profitandClaude Opus 4.7 f48dd2f20b Auditor scaffold: types + Gitea client + policy stub + README 
All-Bun sub-agent that watches open PRs on Gitea, reads ship-claims,
and hard-blocks merges when the code doesn't back the claim. First
commit of N; this is the skeleton. Dynamic/static/inference/kb checks
+ poller land in follow-up commits on this same branch.

- auditor/types.ts — Claim, Finding, Verdict, PrSnapshot shapes
- auditor/gitea.ts — minimal API client (listOpenPrs, getPrDiff,
  postCommitStatus, postReview). Live-proven: returned 0 open PRs
  against our repo (which IS the current state — every commit today
  went to main directly, which is the problem this auditor is meant
  to prevent)
- auditor/policy.ts — stub `assembleVerdict` + severity rules.
  Intentionally conservative defaults: strong claim + zero evidence
  = block, not warn.
- auditor/README.md — how to run + the hard-block mechanism

Workflow discipline change: starting with this branch, no more
direct pushes to main. Every change lands as a PR. When this
auditor is fully built and running, it'll review its own
completion PR — the recursive self-test.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-22 03:26:56 -05:00
profitandClaude Opus 4.7 affab8ac83 Phase 45 slice 2: context7 HTTP bridge for doc drift detection 
Bun bridge on :3900 that wraps context7's public API and exposes the
surface gateway consumes for Phase 45 drift checks. Own port so a
failure here never tips over mcp-server on :3700.

Endpoints:
  GET /health                    status + cache stats
  GET /docs/:tool                resolve tool → library_id → fetch
                                 docs → return descriptor
                                 {snippet_hash, last_updated,
                                 source_url, docs_preview, ...}
  GET /docs/:tool/diff?since=X   compare current snippet_hash to X;
                                 returns {drifted: bool, current,
                                 previous, preview if drifted}
  GET /cache                     debug dump of cached entries

Implementation notes:
- 5 minute in-memory cache (context7 rate-limits by IP; gateway
  drift-checks are the hot caller)
- 1500-token slices from context7 (enough for drift-meaningful
  hash, not so much we hammer their API)
- snippet_hash = SHA-256 prefix (16 hex chars) of fetched content
- Library resolution prefers "finalized" state; falls back to top
  result if none finalized

Verified live against context7.com:
- /health                                  → ok, 0 cache, 300s TTL
- /docs/docker                             → library_id /docker/docs,
                                             title "Docker", hash
                                             475a0396ca436bba, last
                                             updated 2026-04-20
- /docs/docker (again)                     → cache hit, 0.37ms
                                             (5400× speedup)
- /docs/docker/diff?since=stale-hash-0000  → drifted=true, preview
                                             included
- /docs/docker/diff?since=<current hash>   → drifted=false, preview
                                             omitted (honest: no
                                             drift to show)

Not yet wired:
- Gateway consumer (Phase 45 slice 3):
  /vectors/playbook_memory/doc_drift/check/{id} calls this bridge
  and updates DocRef.snippet_hash + doc_drift_flagged_at
- Systemd unit (bridge is manual-start for now, same as bot/)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-22 03:17:17 -05:00
profitandClaude Opus 4.7 2a4b81bf48 Phase 45 (first slice): DocRef + doc_refs field on PlaybookEntry 
Phase J keeps asking for: playbooks know which external docs they
used, get flagged when those docs drift. This commit ships the data
model; context7 bridge + drift check endpoints land in follow-ups.

Added to crates/vectord/src/playbook_memory.rs:
- pub struct DocRef { tool, version_seen, snippet_hash, source_url,
  seen_at } — one external doc reference
- PlaybookEntry.doc_refs: Vec<DocRef> — empty on legacy entries,
  serde default ensures pre-Phase-45 persisted state loads cleanly
- PlaybookEntry.doc_drift_flagged_at: Option<String> — set by the
  (future) drift-check code when context7 reports newer version
- PlaybookEntry.doc_drift_reviewed_at: Option<String> — set by
  human via /resolve endpoint after reviewing the diagnosis
- impl Default for PlaybookEntry — collapses most test-helper
  constructors from 17 explicit fields to 6-9 fields +
  ..Default::default()

Updated SeedPlaybookRequest + RevisePlaybookRequest (service.rs) to
accept optional doc_refs: the seed/revise endpoints already take the
field, downstream drift detection (Phase 45.2) consumes it.

Docs: docs/CONTROL_PLANE_PRD.md gains full Phase 45 spec with gate
criteria, non-goals, and risk notes.

Tests: 51/51 vectord lib tests green (same count as before, field
additions are backward-compat).

Memory: project_doc_drift_vision.md written so this keeps coming
back to the front of mind.

Next slices (same phase): context7 HTTP bridge in mcp-server,
/vectors/playbook_memory/doc_drift/check/{id} endpoint, overview-
model drift synthesis writing to data/_kb/doc_drift_corrections.jsonl,
boost exclusion for flagged+unreviewed entries.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-22 03:14:07 -05:00
profitandClaude Opus 4.7 75a0f424ef Phase 40 (early): Langfuse tracing on /v1/chat — observability recovery 
The lost stack J flagged was partly already present: Langfuse
container has been running 2 days with the staffing project, SDK
installed, mcp-server tracing gw:/* routes. What was missing was
Rust-side /v1/chat emission — the new Phase 38/39 code bypassed
Langfuse entirely.

This commit bridges it. Fire-and-forget HTTP POST to
http://localhost:3001/api/public/ingestion (batch {trace-create +
generation-create}) on every chat call. Non-blocking — spawned
tokio task, response latency unaffected. Trace failures log warn
and drop, never propagate.

Verified end-to-end after restart:
- Log line "v1: Langfuse tracing enabled" at startup
- /v1/chat local (qwen3.5:latest) → v1.chat:ollama trace appears
  with lat=0.41s, 24+6 tokens
- /v1/chat cloud (gpt-oss:120b) → v1.chat:ollama_cloud trace appears
  with lat=1.87s, 73+87 tokens
- mcp-server's existing gw:/log + gw:/intelligence/* traces
  continue to flow into the same project unchanged

Files:
- crates/gateway/src/v1/langfuse_trace.rs (new, 195 LOC) — thin
  client, no SDK. reqwest Basic Auth. ChatTrace payload + event
  serializer. from_env_or_defaults() resolver matches
  mcp-server/tracing.ts conventions (pk-lf-staffing / sk-lf-
  staffing-secret / localhost:3001)
- crates/gateway/src/v1/mod.rs — V1State.langfuse field, emission
  after successful provider call (post-dispatch, pre-usage-update)
- crates/gateway/src/main.rs — resolve + log at startup

Tests: 12/12 green (9 prior + 3 for langfuse_trace: ingestion-batch
serialization, uuid generator uniqueness, env resolver shape).

Recovered piece #1 of 3 from the lost-stack narrative. Still open:
- Langfuse → observer :3800 pipe (Phase 40 mid-deliverable)
- Gitea MCP reconnect in mcp-server/index.ts (Phase 40 late)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-22 03:04:28 -05:00
profit 6316433062 Phase 40 scope: Langfuse + Gitea MCP recovery as named deliverables 
J flagged that a prior version of this stack had Langfuse traces
piping into the observer + Gitea MCP for repo ops — lost. Adding
these as explicit Phase 40 deliverables alongside routing engine
+ Gemini/Claude adapters.

Findings during scope-check:
- Langfuse container is already running (Up 2 days, langfuse:2,
  localhost:3001 healthcheck passes)
- mcp-server/tracing.ts + package.json already have SDK wired
- Credentials pk-lf-staffing / sk-lf-staffing-secret (from env)
- Gitea MCP binary still installed at gitea-mcp@0.0.10

So recovery here is mostly re-connecting existing infra:
1. Add Rust-side Langfuse client for /v1/chat tracing (gateway
   currently bypasses tracing, mcp-server already has it)
2. Wire Langfuse → observer :3800 pipe
3. Register Gitea MCP in mcp-server/index.ts tool list

Each landing as part of Phase 40 when the routing engine ships.
 2026-04-22 03:01:28 -05:00
profitandClaude Opus 4.7 42a11d35cd Phase 39 (first slice): Ollama Cloud adapter on /v1/chat 
Second provider wired. /v1/chat now routes by optional `provider`
field: default "ollama" hits local via sidecar, "ollama_cloud"
(or "cloud") hits ollama.com/api/generate directly with Bearer auth.
Key sourced at gateway startup from OLLAMA_CLOUD_KEY env, then
/root/llm_team_config.json (providers.ollama_cloud.api_key), then
OLLAMA_CLOUD_API_KEY env. Config source matches LLM Team convention.

Shape-identical to scenario.ts::generateCloud — same endpoint, same
body, same Bearer auth. Cloud path bypasses sidecar entirely (sidecar
is local-only by design, mirrors TS agent.ts).

Changes:
- crates/gateway/src/v1/ollama_cloud.rs (new, 130 LOC) — reqwest
  client, resolve_cloud_key(), chat() adapter, CloudGenerateBody /
  CloudGenerateResponse wire shapes
- crates/gateway/src/v1/ollama.rs — flatten_messages_public()
  re-export so sibling adapters reuse the shape collapse
- crates/gateway/src/v1/mod.rs — provider field on ChatRequest,
  dispatch match in chat() handler, ollama_cloud_key on V1State
- crates/gateway/src/main.rs — resolves cloud key at startup,
  logs which source provided it
- crates/gateway/Cargo.toml — reqwest 0.12 with rustls-tls

Verified end-to-end after restart:
- provider=ollama → qwen3.5:latest local (~400ms, Phase 38 unchanged)
- provider=ollama_cloud + model=gpt-oss:120b → real 225-word
  technical response in 5.4s, 313 tokens

Tests: 9/9 green (7 from Phase 38 + 2 new for cloud body serialization
and key resolver shape).

Not in this slice: trait extraction (full Phase 39 scope adds
ProviderAdapter trait + OpenRouter adapter + fallback chain logic).
These land next with Phase 40 routing engine on top.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-22 02:57:42 -05:00
profitandClaude Opus 4.7 8cbbd0ef70 Phase 38 fix: default think=false on /v1/chat 
Live-test caught the Phase 21 thinking-model trap on first call.
qwen3.5 with max_tokens=50 and default think behavior burned all 50
tokens on hidden reasoning; visible content was "". completion_tokens
exactly matching max_tokens was the tell.

Adapter now defaults think: Some(false) matching scenario.ts hot-path
discipline. Callers that want reasoning (overseers, T3+) opt in via
a non-OpenAI `think: true` extension field on the request.

Verified end-to-end after restart:
- "Lakehouse supports ACID and raw data." (5 words, 516ms)
- "tokio\nasync-std\nsmol" (3 Rust crates, 391ms)
- /v1/usage accumulates across calls (2 req / 95 total tokens)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-22 02:50:09 -05:00
profitandClaude Opus 4.7 4cb405bb42 Phase 38: Universal API skeleton — /v1/chat, /v1/usage, /v1/sessions 
First slice of the control-plane pivot. OpenAI-compatible surface
over the existing aibridge → Ollama path. Additive — no existing
routes touched. All 7 unit tests green, release build clean.

What ships:
- crates/gateway/src/v1/mod.rs — router, V1State (ai_client + Usage
  counter), ChatRequest/ChatResponse/Message/UsageBlock types, handlers
  for /chat, /usage, /sessions. OpenAI-compatible field shapes:
  {model, messages[{role,content}], temperature?, max_tokens?, stream?}
- crates/gateway/src/v1/ollama.rs — shape adapter. Flattens messages
  into (system, prompt), calls aibridge.generate, unwraps response
  back into OpenAI /v1/chat shape. Prefers sidecar-reported tokens;
  falls back to chars/4 ceiling estimate matching Phase 21 convention.
- crates/gateway/src/main.rs — one new mod, one .nest("/v1", ...)

Tests (7/7):
- chat_request_parses_openai_shape
- chat_request_accepts_minimal
- usage_counter_default_is_zero
- flatten_separates_system_from_turns
- flatten_concatenates_multiple_system_messages
- flatten_with_no_system_returns_empty_system
- estimate_tokens_chars_div_4_ceiling

Not in this phase (per CONTROL_PLANE_PRD.md): streaming, tool calls,
session state, multi-provider, fallback chain, cost gating. All
land in Phases 39-44.

Next: live-test POST /v1/chat after gateway restart, then migrate
bot/propose.ts off direct sidecar calls to prove the loop end-to-end.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-22 02:47:15 -05:00
profitandClaude Opus 4.7 f44b6b3e6b Control-plane pivot: Phase 38-44 plan + bot scaffold 
Direction shift 2026-04-22: docs/CONTROL_PLANE_PRD.md becomes the
long-horizon architecture target. Existing Lakehouse (docs/PRD.md,
Phases 0-37) is preserved as the reference implementation and first
consumer. New 6-layer architecture:

  L1 Universal API /v1/chat /v1/usage /v1/sessions /v1/tools /v1/context
  L2 Routing & Policy Engine (rules, fallback chains, cost gating)
  L3 Provider Adapter Layer (Ollama + OpenRouter + Gemini + Claude)
  L4 Knowledge + Memory + Playbooks (already built)
  L5 Execution Loop (scenarios + bot/cycle.ts instances)
  L6 Observability + token accounting

Phases 38-44 sequenced with detailed per-phase specs in the PRD.
Current scope: staffing domain (synthetic workers_500k, contracts,
emails, SMS, playbooks). DevOps (Terraform/Ansible) is long-horizon
target — architecture-compatible but not current.

Files added:
- docs/CONTROL_PLANE_PRD.md — 6-layer architecture, Phase 38-44
  sequencing with staffing-first Truth Layer + Validation pipeline
- bot/ — manual-only PR bot scaffold. First consumer test-bed for
  /v1/chat (Phase 38). Mem0-aligned ADD/UPDATE/NOOP apply semantics;
  KB feedback loop reads prior cycles on same gap and injects into
  cloud prompt so bot cycles compound like scenario.ts runs do.
- tests/multi-agent/run_stress.ts — the 6-task diverse stress test
  referenced in the previous commit but missing from its staging

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-22 02:43:31 -05:00
profitandClaude Opus 4.7 5b1fcf6d27 Phase 28-36 body of work 
Accumulated since a6f12e2 (Phase 21 Rust port + Phase 27 versioning):

- Phase 36: embed_semaphore on VectorState (permits=1) serializes
  seed embed calls — prevents sidecar socket collisions under
  concurrent /seed stress load
- Phase 31+: run_stress.ts 6-task diverse stress scaffolding;
  run_e2e_rated.ts + orchestrator.ts tightening
- Catalog dedupe cleanup: 16 duplicate manifests removed; canonical
  candidates.parquet (10.5MB -> 76KB) + placements.parquet (1.2MB ->
  11KB) regenerated post-dedupe; fresh manifests for active datasets
- vectord: harness EvalSet refinements (+181), agent portfolio
  rotation + ingest triggers (+158), autotune + rag adjustments
- catalogd/storaged/ingestd/mcp-server: misc tightening
- docs: Phase 28-36 PRD entries + DECISIONS ADR additions;
  control-plane pivot banner added to top of docs/PRD.md (pointing
  at docs/CONTROL_PLANE_PRD.md which lands in next commit)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-22 02:41:15 -05:00
profitandClaude Opus 4.7 a6f12e2609 Phase 21 Rust port + Phase 27 playbook versioning + doc-sync 
Phase 21 — Rust port of scratchpad + tree-split primitives (companion to
the 2026-04-21 TS shipment). New crates/aibridge modules:

  context.rs       — estimate_tokens (chars/4 ceil), context_window_for,
                     assert_context_budget returning a BudgetCheck with
                     numeric diagnostics on both success and overflow.
                     Windows table mirrors config/models.json.
  continuation.rs  — generate_continuable<G: TextGenerator>. Handles the
                     two failure modes: empty-response from thinking
                     models (geometric 2x budget backoff up to budget_cap)
                     and truncated-non-empty (continuation with partial
                     as scratchpad). is_structurally_complete balances
                     braces then JSON.parse-checks. Guards the degen case
                     "all retries empty, don't loop on empty partial".
  tree_split.rs    — generate_tree_split map->reduce with running
                     scratchpad. Per-shard + reduce-prompt go through
                     assert_context_budget first; loud-fails rather than
                     silently truncating. Oldest-digest-first scratchpad
                     truncation at scratchpad_budget (default 6000 t).

TextGenerator trait (native async-fn-in-trait, edition 2024). AiClient
implements it; ScriptedGenerator test double lets tests inject canned
sequences without a live Ollama.

GenerateRequest gained think: Option<bool> — forwards to sidecar for
per-call hidden-reasoning opt-out on hot-path JSON emitters. Three
existing callsites updated (rag.rs x2, service.rs hybrid answer).

Phase 27 — Playbook versioning. PlaybookEntry gained four optional
fields (all #[serde(default)] so pre-Phase-27 state loads as roots):

  version           u32, default 1
  parent_id         Option<String>, previous version's playbook_id
  superseded_at     Option<String>, set when newer version replaces
  superseded_by     Option<String>, the playbook_id that replaced

New methods:

  revise_entry(parent_id, new_entry) — appends new version, stamps
    superseded_at+superseded_by on parent, inherits parent_id and sets
    version = parent + 1 on the new entry. Rejects revising a retired
    or already-superseded parent (tip-of-chain is the only valid
    revise target).
  history(playbook_id) — returns full chain root->tip from any node.
    Walks parent_id back to root, then superseded_by forward to tip.
    Cycle-safe.

Superseded entries excluded from boost (same rule as retired): filter
in compute_boost_for_filtered_with_role (both active-entries prefilter
and geo-filtered path), rebuild_geo_index, and upsert_entry's existing-
idx search. status_counts returns (total, retired, superseded, failures);
/status JSON reports active = total - retired - superseded.

Endpoints:
  POST /vectors/playbook_memory/revise
  GET  /vectors/playbook_memory/history/{id}

Doc-sync — PHASES.md + PRD.md drifted from git after Phases 24-26
shipped. Fixes applied:

  - Phase 24 marked shipped (commit b95dd86) with detail of observer
    HTTP ingest + scenario outcome streaming. PRD "NOT YET WIRED"
    rewritten to reflect shipped state.
  - Phase 25 (validity windows, commit e0a843d) added to PHASES +
    PRD.
  - Phase 26 (Mem0 upsert + Letta hot cache, commit 640db8c) added.
  - Phase 27 entry added to both docs.
  - Phase 19.6 time decay corrected: was documented as "deferred",
    actually wired via BOOST_HALF_LIFE_DAYS = 30.0 in playbook_memory.rs.
  - Phase E/Phase 8 tombstone-at-compaction limit note updated —
    Phase E.2 closed it.

Tests: 8 new version_tests in vectord (chain-metadata stamping,
retired/superseded parent rejection, boost exclusion, history from
root/tip/middle, legacy default round-trip, status counts). 25 new
aibridge tests (context/continuation/tree_split). Workspace total
145 green (was 120).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-21 17:40:49 -05:00
root 640db8c63c Phase 26 — Mem0 upsert + Letta geo hot cache 
Closes the two remaining 2026-era memory findings. Both are
optimizations per J's framing — not load-bearing, but good data
hygiene + future-proofing at scale.

MEM0 UPSERT (data hygiene):
Before: /seed always appended. A scenario re-running the same
operation on the same day wrote duplicate entries, inflating the
playbook corpus with near-identical rows.

Now: upsert_entry(new) inspects existing non-retired entries and
decides ADD / UPDATE / NOOP:
  ADD     → no matching (operation, day, city, state) tuple, append
  UPDATE  → match exists with different names → merge (union, stable
            order), refresh timestamp, keep original playbook_id so
            citations stay valid
  NOOP    → match exists with identical names → skip, return id

Day-granularity keying on timestamp YYYY-MM-DD means intraday
re-seeds dedup but tomorrow's same-operation is a fresh ADD. Retired
entries don't block new seeds — they're out of scope anyway.

Seed endpoint returns {outcome: {mode, playbook_id, merged_names?},
entries_after}. Append=false retains old replace-all semantics.

5 unit tests pass: first_seed_is_add, identical_reseed_is_noop,
same_day_different_names_updates_and_merges, different_day_same_op_is_add,
retired_entry_doesnt_block_new_seed.

Live verified: three successive seeds with (Alice), (Alice),
(Alice, Bob) left entry count unchanged at 1936 with merged names
{Alejandro, Lauren, Alice, Bob}. Previously would have been 3
appends.

LETTA GEO HOT CACHE (scale primitive):
Added geo_index: HashMap<(city_lower, state_upper), Vec<usize>>
alongside PlaybookMemoryState. Rebuilt on every mutation: set_entries,
retire_one, retire_on_schema_drift, upsert_entry, load_from_storage.

compute_boost_for_filtered_with_role now uses the index for O(1) geo
lookup instead of scanning all entries. At current scale (1.9K) the
scan was sub-ms; at 100K+ the scan becomes the dominant cost. The
hot cache future-proofs without adding an LRU abstraction.

Retired entries excluded from index; valid_until still checked on the
hot path since it can elapse between rebuilds.

Owns cloned PlaybookEntries in the geo_filtered vector so the state
read-lock is released before cosine scoring — avoids lock contention
on the scoring path.

Memory-findings progress: 5 of 5 shipped.
  ✓ Multi-strategy parallel retrieval (Phase 19 refinement)
  ✓ Input normalization + unified /memory/query (Phase 24 TS)
  ✓ Zep validity windows (Phase 25)
  ✓ Mem0 UPSERT (Phase 26 today)
  ✓ Letta geo hot cache (Phase 26 today)

All 18 playbook_memory tests pass.
 2026-04-21 00:24:05 -05:00
root 138592dc56 Spec v2 — all chapters aligned with Phases 19-25 
Full audit pass on devop.live/lakehouse/spec. Five chapters were
stale, one had an outright incorrect line. Scope was bigger than
ch6 alone — J asked "you want to update all" and the honest answer
was yes.

Ch 1 (Repository layout):
- mcp-server row gains /memory/query, /models/matrix, /system/summary,
  observer.ts with :3800 listener
- tests/multi-agent/ row lists all new files: kb.ts, normalize.ts,
  memory_query.ts, gen_scenarios.ts, gen_staffer_demo.ts, and the
  colocated unit tests (kb.test.ts, normalize.test.ts)
- NEW config/ row documents models.json as the 5-tier matrix
- data/ row enumerates the four learning-loop directories:
  _kb/, _playbook_lessons/, _observer/, _chunk_cache/

Ch 3 (Measurement & indexing):
- NEW "Model matrix (Phase 20)" subsection — 5-tier table (T1 hot /
  T2 review / T3 overview / T4 strategic / T5 gatekeeper), per-tier
  primary model, frequency, the think:false mechanical finding
  called out with the 650-token reasoning-budget example
- NEW "Continuation primitive (Phase 21)" paragraph
- NEW "Per-staffer tool_level (Phase 23)" section with full/local/
  basic/minimal mapping and the 46pt fill-rate delta from the 36-run
  demo

Ch 7 (Scale story):
- FIX: playbook_memory growth bullet was claiming "No TTL or merge
  policy" — Phase 25 added retirement via valid_until +
  schema_fingerprint + /retire endpoint. Rewritten to name current
  state (1936 entries, active vs retired split exposed).

Ch 8 (Error surfaces):
- Five new rows added to the failure-mode table:
  * Zero-supply city → cloud rescue (Phase 22 item B) with the
    Gary IN → South Bend IN concrete example
  * LLM truncation → generateContinuable (Phase 21)
  * Schema migration → /vectors/playbook_memory/retire (Phase 25)
  * Observer unreachable → scenario silent-skip + append journal
    survivability

Ch 9 (Per-staffer context):
- NEW "Staffer identity + competence-weighted retrieval (Phase 23)"
  section with the competence_score formula and findNeighbors
  weighted_score
- NEW "Auto-discovered reliable-performer labels" section naming
  Rachel D. Lewis (18 endorsements) and Angela U. Ward (19) as
  concrete output of 36-run demo

Ch 10 (A day in the life):
- Added 17:15 timeline entry — Kim using /memory/query with natural
  language, regex normalizer extracting role/city/count in 0ms
- 17:00 entry updated to mention KB indexing + pathway recommendation
  + observer stream
- 22:00 entry updated to mention detectErrorCorrections nightly scan

Ch 11 (Known limits & non-goals):
- FIX: "playbook_memory compaction" bullet rewritten since retirement
  is now wired; reframed as the honest Mem0 UPDATE/NOOP gap
- Added Letta hot cache deferred item with honest "cheap at 1.9K,
  will bite at 100K" framing
- Added Chunking cache (Phase 21 Rust port) deferred item
- Added Observer → autotune feedback wire deferred item (Phase 26+)

Footer bumped v1 2026-04-20 → v2 2026-04-21 with Phase list.

Verified all updates live on devop.live/lakehouse/spec.
 2026-04-21 00:16:41 -05:00
root e0a843d1a5 Phase 25 — validity windows + playbook retirement 
Addresses the load-bearing memory gap J flagged: playbook entries
had timestamps but no retirement semantic. When a schema migration
changed a column or a seasonal contract ended, stale playbooks kept
boosting candidates silently. Zep 2026-era finding — temporal
validity is the single highest-value memory-hygiene primitive.

SCHEMA (PlaybookEntry gains four optional fields, serde default):
  schema_fingerprint  — SHA-256 over dataset (column, type) tuples at
                        seed time. Missing = legacy entry, never
                        auto-retired on drift.
  valid_until         — RFC3339 hard expiry. compute_boost skips
                        entries past this moment.
  retired_at          — Set by retire_one or retire_on_schema_drift.
                        Retired entries excluded from all boost
                        calculations but kept in journal.
  retirement_reason   — Human-readable: "schema_drift: ...",
                        "expired: ...", "manual: ..."

RETRIEVAL PATH (compute_boost_for_filtered_with_role):
  Before geo+cosine, active_entries filter removes anything retired
  OR past valid_until. Uses chrono::Utc::now() once per call, no per-
  entry clock queries.

NEW METHODS on PlaybookMemory:
  retire_one(playbook_id, reason)
  retire_on_schema_drift(city, state, current_fp, reason) — idempotent,
    scopes by (city, state) so a Nashville migration doesn't touch
    Chicago. Skips legacy entries with no fingerprint.
  status_counts() -> (total, retired, failures)

HTTP ENDPOINTS:
  POST /vectors/playbook_memory/retire
    {playbook_id, reason}                        → retire by id
    {city, state, current_schema_fingerprint, reason} → schema drift
  GET  /vectors/playbook_memory/status
    {total, active, retired, failures}

SEED REQUEST extended with optional schema_fingerprint + valid_until
so the orchestrator (scenario.ts) can pass the current schema hash
when seeding, without a round trip through catalogd.

UNIT TESTS (5/5 pass): retire_one_marks_entry_and_persists,
retired_entries_do_not_boost, expired_valid_until_is_skipped,
schema_drift_retires_mismatched_fingerprints_only,
schema_drift_skips_other_cities.

LIVE VERIFIED: /status on current state = 1936 entries, 43 failures.
POST /retire with a sample playbook_id → "retired":1, /status now
reports active=1935, retired=1.

Memory-findings progress: 3 of 5 shipped.
  ✓ Multi-strategy parallel retrieval (Phase 19 refinement)
  ✓ Input normalization + unified /memory/query (Phase 24 TS)
  ✓ Zep-style validity windows (Phase 25, tonight)
  ⏳ Mem0 UPDATE / DELETE / NOOP ops (dedup same-(op,date) seeds)
  ⏳ Letta working-memory hot cache (not biting at 1.5K entries)
 2026-04-21 00:11:02 -05:00
root 3fb3a60da4 Spec ch6 rewrite — 3 learning paths → 7 + honest gap list 
J flagged the spec out of alignment with what's built. Ch6 now
reflects the full current architecture:

- Path 1 (playbook boost) — formula kept; geo+role prefilter
  refinement called out with measured 14× citation lift
- Path 2 (pattern discovery) — unchanged
- Path 3 (autotune agent) — unchanged
- Path 4 (KB + pathway recommender) — Phase 22, file layout
  documented
- Path 5 (cloud rescue on failure) — Phase 22 item B, verified
  stress_01 example cited
- Path 6 (staffer competence-weighted retrieval) — Phase 23,
  competence_score formula included, cross-staffer auto-
  discovered worker labels (Rachel D. Lewis 18× endorsements)
- Path 7 (observer outcome ingest) — Phase 24, :3800 HTTP
  listener + ops.jsonl append journal

Input normalizer + unified /memory/query surface documented as
the "seamless with whatever input" answer, with the 319ms
natural-language latency number.

Honest gaps kept visible in the spec itself, not hidden:
- Zep validity windows (most load-bearing remaining)
- Mem0 UPDATE/DELETE/NOOP ops
- Letta working-memory hot cache

Live at https://devop.live/lakehouse/spec#ch6 after service
restart. Verified post-deploy: geo+role prefilter, 14× delta,
validity windows gap all present in served HTML.
 2026-04-21 00:03:06 -05:00
root 52561d10d3 Input normalizer + unified memory query — "seamless with whatever input" 
J asked directly: "did we implement our memory findings so that our
knowledge base and our configuration playbook [work] seamlessly with
whatever input they're given?" Honest answer tonight was "one of five
findings shipped, normalizer is the blocker." This closes that gap.

NORMALIZER (tests/multi-agent/normalize.ts):
Accepts structured JSON, natural language, or mixed. Returns canonical
NormalizedInput { role, city, state, count, client, deadline, intent,
confidence, extraction_method, missing_fields } for any downstream
consumer.

Three-tier path:
  1. Structured fast-path — already-shaped input skips LLM
  2. Regex path — "need 3 welders in Nashville, TN" parses without LLM.
     City/state parser tightened to 1-3 capitalized words + "in {city}"
     anchor preference + case-exact full-state-name variants to prevent
     "Forklift Operators in Chicago" being captured as the city name
  3. LLM fallback — qwen3 local with think:false + 400 max_tokens for
     inputs the regex can't handle

Unit tests (tests/multi-agent/normalize.test.ts): 9/9 pass. Covers
structured fast-path, misplacement→rescue intent, state-name→abbrev
conversion, regex extraction from natural language, plural role +
full state name edge case, rescue intent keyword precedence, partial
input reporting missing fields, empty object fallthrough, async/sync
parity on clean inputs.

UNIFIED MEMORY QUERY (tests/multi-agent/memory_query.ts):
One function, five parallel fan-outs, one bundle returned:
  - playbook_workers — hybrid_search via gateway with use_playbook_memory
  - pathway_recommendation — KB recommender for this sig
  - neighbor_signatures — K-NN sigs weighted by staffer competence
  - prior_lessons — T3 overseer lessons filtered by city/state
  - top_staffers — competence-sorted leaderboard
  - discovered_patterns — top workers endorsed across past playbooks
    for this (role, city, state)
  - latency_ms — per-source + total
Every branch is best-effort: one source down doesn't break the bundle.

HTTP ENDPOINT (mcp-server/index.ts):
  POST /memory/query with body {input: <anything>} → MemoryQueryResult
Returns the same shape the TS function does. Typed with types.ts for
future UI consumption.

VERIFIED:
  curl POST /memory/query with structured {role,city,state,count}
    → extraction_method=structured, 10 playbook workers, top score 0.878
  curl POST /memory/query with "I need 3 welders in Nashville, TN"
    → extraction_method=regex (no LLM call), 319ms total, 8 endorsements
      for Lauren Gomez auto-discovered as top Nashville Welder

Honest remaining gaps (documented for next phase):
  - Mem0 ADD/UPDATE/DELETE/NOOP — we still only ADD + mark_failed
  - Zep validity windows — playbook entries have timestamps but no
    retirement semantic
  - Letta working-memory / hot cache — every query scans all 1560
    playbook entries
  - Memory profiles / scoped queries — global pool, no per-staffer
    private subsets

2 of 5 findings now shipped (multi-strategy retrieval in Rust, input
normalization + unified query in TS). The remaining 3 are architectural
additions queued as Phase 25 items — validity windows first since it's
the most load-bearing for long-running systems.
 2026-04-20 23:59:05 -05:00
root b95dd86556 Phase 24 — observer HTTP ingest + scenario outcome streaming 
Closes the gap J flagged: observer wraps MCP:3700, scenarios hit
gateway:3100 directly, observer idle at 0 ops across 3600+ cycles.
Now scenarios POST per-event outcomes to observer's new HTTP ingest
on :3800, observer consumes them alongside MCP-wrapped ops, ERROR_
ANALYZER and PLAYBOOK_BUILDER loops see the full picture.

observer.ts:
- Bun.serve() HTTP listener on OBSERVER_PORT (default 3800):
  GET /health    — basic + ring depth
  GET /stats     — total / success / failure / by_source / recent
                   scenario ops digest
  POST /event    — accept scenario outcome, shape it into ObservedOp
                   with source="scenario" + staffer_id + sig_hash +
                   event_kind + role/city/state + rescue flags
- recordExternalOp() — shared ring-buffer insert so the main analyzer
  + playbook builder don't care where the op came from
- ObservedOp extended with provenance fields

persistOp() FIX — old path POSTed to /ingest/file?name=observed_operations
which REPLACES the dataset (flagged in feedback_ingest_replace_semantics.md).
Every op was silently wiping all prior ops. Replaced with append to
data/_observer/ops.jsonl so the historical trace is durable across
analyzer cycles and process restarts.

scenario.ts:
- OBSERVER_URL env (default http://localhost:3800)
- postObserverEvent() helper with 2s AbortSignal.timeout so observer
  being down doesn't block scenario flow
- Per-event POST after ctx.results.push(result), carrying staffer_id,
  sig_hash (via imported computeSignature), event_kind + role + city
  + state + count + rescue_attempted / rescue_succeeded + truncated
  output_summary

VERIFIED:
  curl POST /event → {"accepted":true,"ring_size":1}
  curl GET /stats → {"total":1,"successes":1,"by_source":{"scenario":1},
    "recent_scenario_ops":[{...staffer_id,kind,role}]}

Final v3 demo leaderboard (9 runs per staffer, cumulative 3 batches):
  James (local):   92.9% fill, 36.8 cites, score 0.775 — RANK 1
  Maria (full):    81.0% fill, 26.2 cites, score 0.727
  Sam (basic):     61.9% fill, 28.2 cites, score 0.640
  Alex (minimal):  59.5% fill, 32.2 cites, score 0.631
Honest finding: Alex has MORE citations than Sam despite NO T3 and NO
rescue. Playbook inheritance alone is firing hardest when overseer is
absent. The 59.5% fill rate (up from 0% when qwen2.5 was executor)
proves cloud-exec + playbook inheritance is the floor the architecture
delivers.

Local gpt-oss:20b T3 outperforms cloud gpt-oss:120b T3 by 12pt fill
rate on this workload — cloud overseer paying latency+variance for
no measurable gain, worth flagging in next models.json tune.
 2026-04-20 23:49:30 -05:00
root 137aed64fb Coherence pass — PRD/PHASES updates, config snapshot wired, unit tests 
J flagged the audit: "make sure everything flows coherently, no
pseudocode or unnecessary patches or ignoring any particular part of
what we built." This is that pass.

PRD.md updates:
- Phase 19 refinement block — geo-filter + role-prefilter WIRED with
  citation density numbers (0.32 → 1.38, and 2 → 28 on same scenario).
- Phase 20 rewrite — mistral dropped, qwen3.5 + qwen3 local hot path,
  think:false as the key mechanical finding, kimi-k2.6 upgrade path.
- Phase 21 status block — think plumbing + cloud executor routing
  added after original commit.
- Phase 22 item B (cloud rescue) — pivot sanitizer, rescue verified
  1/3 on stress_01.
- Phase 23 NEW — staffer identity + tool_level + competence-weighted
  retrieval + kb_staffer_report. Auto-discovered worker labels called
  out with real numbers (Rachel Lewis 12× across 4 staffers).
- Phase 24 NEW — Observer/Autotune integration gap DOCUMENTED, not
  fixed. Observer has been idle at 0 ops for 3600+ cycles because
  scenarios hit gateway:3100 directly, bypassing MCP:3700 which the
  observer wraps. This is the honest "we're not using it in these
  tests" signal J surfaced. Fix deferred; gap visible now.

PHASES.md:
- Appended Phases 20-23 as checked, Phase 24 as unchecked gap.
- Updated footer count: 102 unit tests across all layers.
- Latest line updated with 14× citation lift + 46.4pt tool-asymmetry
  finding.

scenario.ts:
- snapshotConfig() was defined but never called. Now fires at every
  scenario start with a stable sha256 hash over the active model set +
  tool_level + cloud flags. config_snapshots.jsonl finally populates,
  which the error_corrections diff path needs to work correctly.

kb.test.ts (new): 4 signature invariant tests — stability across
unrelated fields (date, contract, staffer), sensitivity to role/city/
count changes, digest shape. All pass under `bun test`.

service.rs: 6 Rust extractor tests for extract_target_geo +
extract_target_role — basic, missing-state-returns-none, word
boundary (civilian != city), multi-word role, absent role, quoted
value parse. All pass under `cargo test -p vectord --lib extractor_tests`.

Dangling items now honestly documented rather than silently pending:
- Chunking cache (config/models.json SPEC, not wired) — flagged
- Playbook versioning (SPEC, not wired) — flagged
- Observer integration (WIRED but disconnected) — new Phase 24
 2026-04-20 23:29:13 -05:00
root ad0edbe29c Cloud kimi-k2.5 executor for weak tiers + multi-strategy playbook retrieval 
Two coupled changes from the 2026 agent-memory research + tool
asymmetry findings.

SCENARIO (weak-tier cloud substitute):
qwen2.5 collapsed to 0/14 across the basic/minimal tool_levels.
Replace with cloud kimi-k2.5 on Ollama Cloud — same family as k2.6
(pro-tier locked today, on J's upgrade path). Plumb cloud flag
through ACTIVE_EXECUTOR_CLOUD / ACTIVE_REVIEWER_CLOUD into
generateContinuable so executor/reviewer can route to cloud when
tool_level requires. think:false supported by Kimi family.

Tool level mapping (revised):
  full     — qwen3.5 local + qwen3 local + cloud gpt-oss:120b T3 + rescue
  local    — qwen3.5 local + qwen3 local + local gpt-oss:20b T3 + rescue
  basic    — kimi-k2.5 cloud + qwen3 local + local T3, no rescue
  minimal  — kimi-k2.5 cloud + qwen3 local, no T3, no rescue.
             Playbook inheritance alone on the decision path.

This is the honest version of J's "minimal tools still works via
inheritance" hypothesis — with the executor no longer broken at the
tokenizer level, we can actually measure whether playbook retrieval
substitutes for missing overseers.

PLAYBOOK_MEMORY (multi-strategy retrieval):
Zep / Mem0 research shows multi-strategy rerank (semantic + keyword +
graph + temporal) outperforms single-strategy cosine. Lakehouse now
has a two-tier:

  1. Exact (role, city, state) match: skip cosine, assign similarity=1.0,
     take up to top_k/2+1 slots. These are identity-class neighbors —
     the strongest possible signal.
  2. Cosine fallback within the same (city, state) but different role:
     fills remaining slots.

Exposed as compute_boost_for_filtered_with_role(target_geo, target_role).
Backwards-compatible: compute_boost_for_filtered forwards with role=None
so existing callers keep their current behavior.

Service.rs wires both: extract_target_geo and extract_target_role pull
from the executor's SQL filter. grab_eq_value is factored out of
extract_target_geo so both lookups share one parser. Diagnostic log
now prints target_role alongside target_geo for every hybrid_search:

  playbook_boost: boosts=88 sources=39 parsed=39 matched=5
    target_geo=Some(("Nashville", "TN")) target_role=Some("Welder")

Verified: Nashville Welder query returns 5/10 boosted workers in
top_k with clean role+geo provenance.

Research sources: atlan.com Agent Memory Frameworks 2026, Mem0 paper
(arxiv 2504.19413), Zep/Graphiti LongMemEval comparison, ossinsight
Agent Memory Race 2026.

kimi-k2.6 on current key returns 403 — pro-tier upgrade required.
kimi-k2.5 is the substitute today; swap to k2.6 by renaming one line
in applyToolLevel once the subscription lands.
 2026-04-20 23:20:07 -05:00
root 5e89407939 Phase 23 refinement — per-staffer tool_level variance 
Staffer.tool_level now controls which subsystems a specific run gets:

  full     — qwen3.5 + qwen3 + cloud T3 + cloud rescue
  local    — qwen3.5 + qwen3 + local gpt-oss:20b T3 + rescue
  basic    — qwen2.5 + qwen2.5 + local T3, no rescue
  minimal  — qwen2.5 + qwen2.5, NO T3, NO rescue. Playbook
             inheritance only.

applyToolLevel() mutates module-scoped ACTIVE_* slots each run from the
env defaults, so prior staffer's overrides never leak. Hot-path code
reads ACTIVE_EXECUTOR / ACTIVE_REVIEWER / ACTIVE_T3_DISABLED /
ACTIVE_OVERVIEW_CLOUD / ACTIVE_RETRY_ON_FAIL instead of the baked
constants.

The architectural question this answers: does playbook_memory
inheritance carry enough knowledge to let a weakly-tooled coordinator
still produce usable outcomes? "Minimal" Alex runs qwen2.5 exec + no
reviewer overseer + no cloud rescue. If Alex still fills events at a
reasonable rate, the playbook system is the real knowledge carrier —
the senior stack is nice-to-have, not the sine qua non.

Demo personas mapped:
  Maria (senior, 48mo, full)
  James (mid, 14mo, local)
  Sam (junior, 4mo, basic)
  Alex (trainee, 1mo, minimal)

Same 3 contracts (Nashville downtown, Joliet warehouse, Indianapolis
assembly) across all four → 12 runs. KB + kb_staffer_report.py
leaderboard already wired; competence_score will now reflect real tool
asymmetry instead of LLM sampling variance.
 2026-04-20 22:50:05 -05:00
root 6b71c8e9b2 Phase 23 — contract terms + staffer identity + competence-weighted retrieval 
Matrix-index the "who handled this" dimension so top staffers become
the training signal and juniors inherit their playbooks automatically
via the boost pipeline. Auto-discovered indicators emerge from
comparing trajectories across staffers on similar contracts — that was
always the architectural point; this wires the last piece.

ContractTerms:
- deadline, budget_total_usd, budget_per_hour_max, local_bonus_per_hour,
  local_bonus_radius_mi, fill_requirement ("paramount" | "preferred")
- Attached to ScenarioSpec, propagated into T3 checkpoint + cloud
  rescue prompts so cloud reasons about trade-offs (pivot within bonus
  radius first; respect per-hour cap; split across cities when
  fill_requirement=paramount).

Staffer:
- {id, name, tenure_months, role: senior|mid|junior|trainee}
- On ScenarioSpec; logged at scenario start; attached to KB outcome
- Recomputed StafferStats written to data/_kb/staffers.jsonl after
  every run: total_runs, fill_rate, avg_turns, avg_citations,
  rescue_rate, competence_score.
- Competence formula: 0.45*fill_rate + 0.20*turn_efficiency +
  0.20*citation_density + 0.15*rescue_rate. Normalized to 0..1.

findNeighbors now returns weighted_score = cosine × best_staffer_competence
(floored at 0.3 so high-similarity low-competence neighbors still
surface). pathway_recommender prompt shows the top staffer's identity
so cloud knows WHOSE playbook it's synthesizing from.

Demo infrastructure:
- tests/multi-agent/gen_staffer_demo.ts: 4 personas (Maria senior,
  James mid, Sam junior, Alex trainee) × 3 contracts (Nashville Welder,
  Joliet Warehouse, Indianapolis Assembly). 12 scenarios total.
- scripts/run_staffer_demo.sh: runs the 12 sequentially with
  LH_OVERVIEW_CLOUD=1. Post-run calls kb_staffer_report.py.
- scripts/kb_staffer_report.py: leaderboard + cross-staffer worker
  overlap (names endorsed by ≥2 staffers → auto-discovered high-value
  workers). Top vs bottom differential.

gen_scenarios.ts (Phase 22 generator) also now emits contract terms
on 70% of generated specs — future KB batches populate with realistic
constraint patterns instead of bare role+city+count.

Stress scenario from item A intentionally NOT the production test.
Real staffing has constraints; Nashville contract + staffer demo is
the honest test of whether the architecture produces measurable
differential between coordinator skill levels.

Demo batch launched — 12 runs × ~3min each ≈ 40min unattended. Report
emitted after batch.
 2026-04-20 22:16:09 -05:00
root a7fc8e2256 Item B — cloud-rescue retry on event failure 
When a scenario event fails (drift abort or other error) and
LH_RETRY_ON_FAIL is on (default when cloud T3 is enabled), ask cloud
for a concrete pivot — new city, role, or count — then re-run the
event with the remediation's fields. Capped at 1 retry per event so a
genuinely-impossible scenario can't burn budget.

requestCloudRemediation(event, result):
- Feeds the same diagnostic bundle T3 checkpoints get (SQL filters,
  row counts, SQL errors, reviewer drift reasons, gap signals).
- Prompt demands structured JSON: {retry, new_city, new_role,
  new_count, rationale}.
- Cloud is instructed to pivot to NEAREST alternate city when
  zero-supply detected, broaden role when uniquely scarce, reduce
  count when clearly unachievable, or return retry=false when no
  pivot seems viable.

EventResult additions:
- retry_attempt, retry_remediation (with rationale + cloud_model +
  duration), retry_result (full inner result shape), original_event.
- If retry succeeded, it becomes the primary result and original_event
  preserves what was attempted first. If retry also failed, the
  primary stays the failure and retry is recorded alongside.

Sanitizer on cloud output: model sometimes emits "Hammond, IN" in
new_city with "IN" in a non-existent new_state field, producing
"Hammond, IN, IN" downstream. Split new_city on comma, take first
token as city, extract state if present after the comma. Original
event's state is the fallback.

VERIFIED on stress_01.json with LH_OVERVIEW_CLOUD=1:
  Without rescue (item A baseline):  1/5 events ok
  With rescue (item B):              3/5 events ok
Gary IN misplacement: drift → cloud proposed South Bend IN → retry
filled 1/1. Rationale stored in retry_remediation for forensics.

Known limits surfaced (future work):
- City-field mangling failed one rescue before the sanitizer landed;
  next run will use the fix.
- Cloud picks alternate cities without knowing ground-truth supply.
  Flint → Saginaw pivoted but Saginaw also had sparse Welders.
  Future: expose a /vectors/supply-estimate endpoint cloud can consult
  before proposing a pivot.
 2026-04-20 22:01:45 -05:00
root c21b261877 Item A — stress scenario + enriched T3 diagnostic prompt 
Proves cloud passthrough works end-to-end AND fixes the diagnostic
quality problem that first run surfaced.

STRESS SCENARIO (tests/multi-agent/scenarios/stress_01.json):
Five genuinely hard events with varied failure modes:
- Gary, IN 5× Electrician: ZERO supply (city not in workers_500k)
- Peoria, IL 8× Safety Coordinator: scarce role, initial pool only 5
- Flint, MI 3× Welder: ZERO supply
- Grand Rapids, MI 4× Tool & Die Maker: scarce but solvable
- Gary, IN 1× Electrician misplacement: repeats event 1's impossibility

FIRST RUN (stress v1) — cloud passthrough works, diagnosis vague:
  T3 checkpoint: "Potential drift flags for upcoming role"
  Lesson: "Before dispatching, query pool status. Update turn counter..."
Generic tactical advice that doesn't address the real problem.
Root cause: T3 prompt only saw outcome summary, not the raw
SQL/pool/drift signals the executor had in its log.

DIAGNOSTIC FIX:
- Added LogEntry[] `sharedLog` parameter to runAgentFill so the caller
  retains the trace even when runAgentFill throws drift-abort.
- EventResult gained `diagnostic_log` field populated on both OK and
  FAIL paths.
- extractDiagnostics() pulls SQL filters, hybrid_search row counts,
  SQL errors, and reviewer drift notes from the log.
- Checkpoint prompt now includes FAILURE FORENSICS block for failed
  events: SQL filters attempted, row counts, errors, drift reasons,
  and an explicit teaching note about zero-supply detection.
- Cross-day lesson prompt flags each event with [ZERO-SUPPLY: pivot
  city needed] tag when drift reasons mention "no match"/"no
  candidates"/"0 rows". PRIORITY clause in the prompt tells the model
  its lesson MUST name alternate cities when that tag appears.

SECOND RUN (stress v2 with enriched prompt) — cloud diagnosis sharp:
  T3 after Flint: risk="Zero candidate supply for Welder in Flint"
                  hint="search Welder×3 in Saginaw, MI (≈30 mi) or
                        expand role to Metal Fabricator"
  T3 after Gary:  risk="Zero supply for Electrician in Gary, IN"
                  hint="Pivot to Chicago, IL (≈40 min); broaden to
                        Electrical Technician within 60 min radius"
  Lesson: specific, per-city, with distances, role-broadening
  fallback, and pre-loading strategy — actionable for item B retry.

Cloud 120b call latencies consistent: 4.8-8.0s per prompt. Cloud
passthrough proven under stress.

Fill outcomes unchanged (1/5 — correct rejection of three impossible
events + one propagating JSON emission edge case on retry pivot
reasoning). The knowledge to rescue them now exists in the lesson;
item B wires the retry.
 2026-04-20 21:54:29 -05:00
root a663698571 Item 3 — geo-filtered playbook boost; diagnostic logging 
ROOT CAUSE (found via instrumentation, not hunch):
After a 20-scenario corpus batch, only 6/40 successful (role, city)
combos ever triggered playbook_memory citations on subsequent runs.
Added `playbook_boost:` tracing::info! line in vectord::service to log
boost map size vs candidate pool vs match count. One query revealed:

  boosts=170 sources=50 parsed=50 matched=0

170 endorsed workers came back from compute_boost_for — but zero were
in the 50-candidate Toledo pool. The boost map was pulling globally-
ranked semantic neighbors (top-100 playbooks across ALL cities),
dominated by Kansas City / Chicago / Detroit forklift playbooks the
Toledo SQL filter would never admit. The mechanism was correct at the
per-playbook level; the problem was pool intersection.

FIX (surgical, not cap-tuning):
- playbook_memory::compute_boost_for_filtered(): accepts optional
  (city, state) filter. When set, skips playbooks from other geos
  BEFORE cosine-ranking, so top-k is within the target city.
- Backwards-compatible: compute_boost_for() calls the filtered variant
  with None — existing callers unchanged.
- service::hybrid_search(): extracts target (city, state) from the
  executor's SQL filter via a small parser (extract_target_geo),
  passes to compute_boost_for_filtered.

VERIFIED:
  Before fix: boosts=170 sources=50 parsed=50 matched=0   (0% hit)
  After fix:  boosts=36  sources=50 parsed=50 matched=11  (22% hit)
Top-k=10 now has 7/10 boosted workers with 2-3 citations each.
Boost values 0.075-0.113 on cosine scores 0.67-0.74 — meaningful
reorder without saturation.

scripts/kb_measure.py:
Aggregator that reads data/_kb/*.jsonl and playbooks/*/results.json,
reports fill rate, citation density, recommender confidence trend,
and zero-citation-ok combos (item 3 target signal). Used to measure
before/after on bigger batches.

Diagnostic logging stays — the class of "boosts computed but not
matched" bug can recur if the SQL filter format ever drifts, and
without the counter it's invisible. Every hybrid_search with
use_playbook_memory=true now logs its boost stats.
 2026-04-20 21:35:04 -05:00
root 330cb90f99 Lift k cap, drop ornamental reason field, scenario generator 
ITEM 1 — k CAP + REASON FIELD
The hybrid_search default k was hard-coded to 10. For multi-fill events
(5× expansion, 4× emergency) that's pool=10 → propose 5-of-10, half
the candidates become the answer with no room for rejection. Executor
prompt now instructs k to scale with target_count: k = max(count*5, 20),
cap 80. Default helper bumped 10 → 20.

Fill.reason dropped from required to optional. Nothing downstream ever
consumed it — resolveWorkerIds, sealSale, retrospective all use
candidate_id and name. Models loved to write 100-150 char justifications
per fill; on 4+ fills that blew the JSON budget before the structure
closed. Test 1 run result after this change: FIRST EVER 5/5 on the
Riverfront Steel scenario, 13 total turns across 5 events. The event
that failed last run (emergency 4×Loader with truncated reason-field
continuation) now clears in 2 turns.

Progression:
  mistral baseline:                  0/5
  qwen3.5 + continuation + think:false: 4/5
  qwen3.5 + k=20 + no-reason:        5/5 ✓

ITEM 2 — SCENARIO GENERATOR (NOT YET TESTED E2E)
tests/multi-agent/gen_scenarios.ts emits N deterministic ScenarioSpecs
with varied clients (15 companies), cities (20 Midwest cities known
to exist in workers_500k), role mixes (14 industrial staffing roles,
weighted realistic), and event sequences. Each gets a unique sig_hash
so the KB populates with distinct neighbor signatures.

scripts/run_kb_batch.sh runs all generated specs sequentially against
scenario.ts, logs per-scenario outcomes, and reports KB state at the
end. Each run takes ~2-4min; 20-30 scenarios = 1-2hr unattended.

Next: test the generator+batch on a small N (3-5) to verify KB
populates correctly and pathway recommendations start getting neighbor
signal instead of cold-starts. Then item 3 (Rust re-weighting of
hybrid_search by playbook_memory success).
 2026-04-20 20:31:34 -05:00
root 9c1400d738 Phase 22 — Internal Knowledge Library (KB) 
Meta-layer over Phase 19 playbook_memory. Phase 19 answers "which
WORKERS worked for this event"; KB answers "which CONFIG worked for
this playbook signature" — model choice, budget hints, pathway notes,
error corrections.

tests/multi-agent/kb.ts:
- computeSignature(): stable sha256 hash of the (kind, role, count,
  city, state) tuple sequence. Same scenario shape → same sig.
- indexRun(): extracts sig, embeds spec digest via sidecar, appends
  outcome record, upserts signature to data/_kb/signatures.jsonl.
- findNeighbors(): cosine-ranks the k most-similar signatures from
  prior runs for a target spec.
- detectErrorCorrections(): scans outcomes for same-sig fail→succeed
  pairs, diffs the model set, logs to error_corrections.jsonl.
- recommendFor(): feeds target digest + k-NN neighbors + recent
  corrections to the overview model, gets back a structured JSON
  recommendation (top_models, budget_hints, pathway_notes), appends
  to pathway_recommendations.jsonl. JSON-shape constrained so the
  executor can inherit it mechanically.
- loadRecommendation(): at scenario start, pulls newest rec matching
  this sig (or nearest).

scenario.ts:
- Reads KB recommendation at startup (alongside prior lessons).
- Injects pathway_notes into guidanceFor() executor context.
- After retrospective, indexes the run + synthesizes next rec.

Cold-start behavior: first run with no history writes a low-confidence
"no prior data" rec so the signal that something was attempted is
captured. Second run gets "low confidence, 0 neighbors" until a third
distinct sig gives the embedder something to compare against — hence
the upcoming scenario generator.

VERIFIED:
- data/_kb/ populated after one scenario run: 1 outcome (sig=4674…,
  4/5 ok, 16 turns total), 1 signature, 2 recs (cold + post-run).
- Recommendation JSON-parsed cleanly from gpt-oss:20b overview model.

PRD Phase 22 added with file layout, cycle description, and the
rationale for file-based MVP → Rust port progression that matches
how Phase 21 primitives shipped.

What's NOT here yet (batched follow-ups per J's request, tested
between each):
- Lift the k=10 hybrid_search cap to adaptive k=max(count*5, 20)
- Scenario generator to bulk-populate KB with varied signatures
- Rust re-weighting: push playbook_memory success signal INTO
  hybrid_search scoring, not just post-hoc boost
 2026-04-20 20:27:12 -05:00
root 0c4868c191 qwen3.5 executor + continuation primitive + think:false 
Three coupled fixes that together turned the Riverfront Steel scenario
from 0/5 (mistral) to 4/5 (qwen3.5) with T3 flagging real staffing
concerns rather than linter advice.

MODEL SWAP
- Executor: mistral → qwen3.5:latest (9.7B, 262K ctx, thinking).
  mistral's decoder emitted malformed JSON on complex SQL filters
  regardless of prompt; J called it — stop using mistral.
- Reviewer: qwen2.5 → qwen3:latest (40K ctx)
- Applied to scenario.ts, orchestrator.ts, network_proving.ts,
  run_e2e_rated.ts

CONTINUATION PRIMITIVE (agent.ts)
- generateContinuable(): empty-response → geometric backoff retry;
  truncated-JSON → continue from partial as scratchpad; bounded by
  budget cap + max_continuations. No more "bump max_tokens until it
  stops truncating" tourniquet.
- generateTreeSplit(): map-reduce for oversized input corpora with
  running scratchpad digest, reduce pass for final synthesis.
- Empty text no longer throws — it's a signal to continuable that
  thinking ate the budget.

think:false FOR HOT PATH
- qwen3.5 burned ~650 tokens of hidden thinking for trivial JSON
  emission. For executor/reviewer/draft: think:false. For T3/T4/T5
  overseers: thinking stays on (that's the point).
- Sidecar generate endpoint accepts `think` bool, passes through to
  Ollama's /api/generate.

VERIFIED OUTCOMES
Riverfront Steel 2026-04-21, qwen3.5+continuable+think:false:
  08:00 baseline_fill  3/3  4 turns
  10:30 recurring      2/2  3 turns (1 playbook citation)
  12:15 expansion      0/5  drift-aborted (5-fill orchestration
                            problem, separate work)
  14:00 emergency      4/4  3 turns (1 citation)
  15:45 misplacement   1/1  3 turns
  → T3 caught Patrick Ross double-booking across events
  → T3 flagged forklift cert drift on the event that failed
  → Cross-day lesson proposed "maintain buffer of ≥3 emergency
    candidates, pre-fetch certs for expansion, booking system
    cross-check" — real staffing advice, not generic linter output

PRD PHASE 21 rewritten to reflect the actual primitive shape (two-
call map-reduce with scratchpad glue) instead of the tourniquet
approach originally documented. Rust port queued for next sprint.

scripts/ab_t3_test.sh: A/B harness that chains B→C→D runs and emits
tests/multi-agent/playbooks/ab_scorecard.json.
 2026-04-20 20:19:02 -05:00
root 6e7ca1830e Phase 21 foundation — context stability + chunking pipeline 
PRD: add Phase 20 (model matrix, wired) and Phase 21 (context stability,
partial). Phase 21 exists because LLM Team hit this exact wall — running
multi-model ranking on large context silently truncated, rankings
degraded, no pipeline caught it. The stable answer: every agent call
goes through a budget check against the model's declared context_window
minus safety_margin, with a declared overflow_policy when the check
fails.

config/models.json:
- context_window + context_budget per tier
- overflow_policies block: summarize_oldest_tool_results_via_t3,
  chunk_lessons_via_cosine_topk, two_pass_map_reduce,
  escalate_to_kimi_k2_1t_or_split_decision
- chunking_cache spec (data/_chunk_cache/, corpus-hash keyed)

agent.ts:
- estimateTokens() chars/4 biased safe ~15%
- CONTEXT_WINDOWS table (fallback; prod reads models.json)
- assertContextBudget() — throws on overflow with exact numbers, can
  bypass with bypass_budget:true for callers with their own policy
- Wired into generate() and generateCloud() so EVERY call is checked

scenario.ts:
- T3 lesson archive to data/_playbook_lessons/*.json (the old
  /vectors/playbook_memory/seed path was silently failing with HTTP 400
  because it requires 'fill: Role xN in City, ST' operation shape)
- loadPriorLessons() at scenario start — filters by city/state match,
  date-sorted, takes top-3
- prior_lessons.json archived per-run (honest signal for A/B)
- guidanceFor() injects up to 2 prior lessons (≤500 chars each) into
  the executor's per-event context
- Retrospective shows explicit "Prior lessons loaded: N" line

Verified: mistral correctly rejects a 150K-char prompt (7532 tokens
over), gpt-oss:120b accepts it with 90K headroom. The enforcement is
in-band on every call now, not an afterthought.

Full chunking service (Rust) remains deferred to the sprint this feeds:
crates/aibridge/src/budget.rs + chunk.rs + storaged/chunk_cache.rs
 2026-04-20 19:34:44 -05:00
root 03d723e7e6 Model matrix — 5 tiers, local hard workers + cloud overseers 
config/models.json is the authoritative catalog. Hot path (T1/T2) stays
local; cloud is consulted only for overview (T3), strategic (T4), and
gatekeeper (T5) calls. J named qwen3.5 + newer models (minimax-m2.7,
glm-5, qwen3-next) specifically — all mapped with real reachable IDs
verified against ollama.com/api/tags.

Tier shape:
- t1_hot     mistral + qwen2.5 local       — 50-200 calls/scenario
- t2_review  qwen2.5 + qwen3 local         — 5-14 calls/event
- t3_overview gpt-oss:120b cloud           — 1-3 calls/scenario
- t4_strategic qwen3.5:397b + glm-4.7      — 1-10 calls/day
- t5_gatekeeper kimi-k2-thinking           — 1-5 calls/day, audit-logged

Rate budgets are declared in-config — Ollama Cloud paid tier is generous
but we cap overview/strategic/gatekeeper so no single rogue scenario can
blow the day's quota.

Experimental rotation list wired but disabled by default. When enabled,
T4 randomly routes 10% of calls to a rotating minimax/GLM/qwen-next/
deepseek/nemotron/cogito/mistral-large candidate, logs comparisons, and
auto-promotes after 3 rotations of wins.

Playbook versioning SPEC embedded under `playbook_versioning` key: every
seed gets version + parent_id + retired_at + architecture_snapshot, so
when a schema migration breaks a playbook we can pinpoint which change
retired it. Implementation flagged for next sprint (touches gateway +
catalogd + mcp-server) — not wired here.

- scenario.ts now loads config/models.json at init, env vars still override
- mcp-server exposes /models/matrix read-only so UI can render it
 2026-04-20 19:24:41 -05:00
root e4ae5b646e T3 overview tier — mid-day checkpoints + cross-day lesson 
Hot path (T1/T2) stays mistral + qwen2.5. The new T3 tier runs a
thinking model SPARINGLY — after every misplacement, every N-th event
(default N=3), and once post-scenario for the cross-day lesson.

- agent.ts: generateCloud() for Ollama Cloud (gpt-oss:120b etc). Uses
  the same /api/generate shape; thinking field is discarded.
- scenario.ts: runOverviewCheckpoint + runCrossDayLesson. Outputs land
  in checkpoints.jsonl and lesson.md. Lesson also seeds playbook_memory
  under operation "cross-day-lesson-{date}" — future runs pick it up
  through the existing similarity boost.
- Env knobs: LH_OVERVIEW_CLOUD=1 routes T3 to cloud, LH_OVERVIEW_MODEL
  overrides (default gpt-oss:20b local, gpt-oss:120b cloud),
  LH_T3_CHECKPOINT_EVERY controls cadence, LH_T3_DISABLE=1 turns it off.

Why this shape: prior feedback_phase19_seed_text.md warned that verbose
seeds dilute the embedding and silently kill the boost. T3's rich prose
goes to lesson.md; the embedded "approach" + "context" stay terse.

Verified end-to-end: local 20b checkpoint 10.9s, lesson 4.0s; cloud
120b lesson 3.7s. Cloud output is both faster AND more specific than
local (sequenced, tactical, logging advice included).
 2026-04-20 19:21:45 -05:00
root 0ff091c173 Honesty fixes — no hard-coded counts, dynamic sample CSV 
- generateSampleRosterCSV(): 120-180 randomized rows per call, timestamp-prefixed IDs (no dedup on re-upload, no static 25 row lie)
- /system/summary: truth via SQL COUNT(*), surfaces manifest_drift (caught candidates: manifest 100K, actual 1K)
- search.html: loadSystemSummary() hydrates live counts; removed hard-coded 500K strings
- MCP tool description: "candidates (100K)" → "candidates (1K)", added "workers_500k (500K)"
 2026-04-20 19:07:47 -05:00
root af3856b103 Rate/margin awareness: implied pay rate per worker, bill rate per contract 
Closes one of the Path 1 trust-break gaps. The scenario we kept flagging:
recruiter calls the system's top pick, worker quotes $35/hr, contract
pays $28/hr. First broken call kills the demo. This fixes it.

Heuristic (no schema change, derived at query time):
- Per worker: implied_pay_rate = role_base + (reliability × 4) + archetype_bump
  role_base: Electrician $28, Welder $26, Machine Op $24, Maint $26,
    Forklift Op $20, Loader $17, Warehouse Assoc $17, Quality Tech $23,
    Production Worker $18 ...
  archetype bump: specialist +4, leader +3, reliable +1, else 0
- Per contract: implied_bill_rate = role_base × 1.4
  (40% markup — industry norm: pay + overhead + insurance + margin)
- Worker is 'over_bill_rate' when implied_pay_rate > contract's bill_rate
  on a candidate-by-candidate basis

Backend (mcp-server/index.ts):
- ROLE_BASE_PAY_RATE + BILL_MARKUP constants
- impliedPayRate(worker), impliedBillRate(role) functions
- parseWorkerChunk() extracts role/reliability/archetype from vector text
- enrichWithRates() attaches implied_pay_rate on every /vectors/hybrid
  source response. Called from /search and /intelligence/permit_contracts.
- /search accepts optional max_pay_rate number — if set, filters out
  workers above that rate and reports pay_rate_filtered_out count.
- /intelligence/permit_contracts returns implied_bill_rate per contract
  AND over_bill_rate boolean per candidate.

Frontend (search.html):
- Live Contracts cards show 'bill rate: $X/hr' under the headcount line
- Each candidate shows 'pay $X/hr' in the sub-line; red 'Over bill rate'
  chip next to name when their pay exceeds the contract's bill rate
  (hover reveals the exact numbers and why it's flagged)
- Main 'Search all workers' results now include 'pay $X/hr' in the
  why-text (computeImpliedPayRate mirrored client-side to match Bun)

End-to-end verified live:
- Masonry Work permit, bill_rate $25.20/hr
  Kathleen M. Gutierrez pay $25.56/hr → 🔴 OVER
  Melissa C. Rivera pay $20.88/hr → 🟢 OK
- /search with max_pay_rate:32 filtered out 1 Toledo Welder above $32
- Main search shows 'pay $28.64/hr' in each result row

When real ATS data replaces synthetic workers_500k, same UI — the
client's real pay_rate column substitutes for the heuristic.
 2026-04-20 18:56:51 -05:00
root a117ae8b38 Workspace UI — surface Phase 8.5 per-contract state + handoff 
Phase 8.5 was fully built on the Rust side (WorkspaceManager with
create/handoff/search/shortlist/activity/get/list, persisted to
object storage, zero-copy handoff between agents). Nothing surfaced
it in the recruiter UI. This page closes that gap.

/workspaces — split-pane UI:

Left: scrollable list of all workspaces, sorted by updated_at.
  Each card shows name, tier pill (daily/weekly/monthly/pinned),
  current owner, count of shortlisted candidates + activity events.

Right: selected workspace detail with five sections:
  1. Header — name, tier, owner, created/updated dates, description,
     previous-owners audit trail (each handoff is preserved)
  2. Actions row — Hand off, Shortlist candidate, Save search, Log activity
  3. Shortlist — candidates flagged with dataset + record_id + notes
  4. Saved searches — named SQL queries the staffer wants to rerun
  5. Activity — chronological (newest first) log of what happened

Four modals for the add/edit actions (create, handoff, shortlist,
save-search, log-activity). All forms POST through the existing
/api/* passthrough to the gateway's /workspaces/* routes.

End-to-end verified live:
  1. Sarah creates 'Demo: Toledo Week 17' workspace
  2. Shortlists Helen Sanchez (W500K-4661) with notes about prior endorsements
  3. Logs activity: 'called — Helen confirmed Tuesday 7am shift'
  4. Hands off to Kim with reason 'end of shift'
  5. Kim opens the workspace: owner=kim, previous_owners=[{sarah→kim}],
     sees all 3 prior events + the shortlisted Helen
     — no data copy, pointer swap only (Phase 8.5 design)

Security: all dynamic content built via el(tag,cls,text) DOM helper.
Zero innerHTML on API-derived strings. Modal close-on-backdrop-click
is guarded to the backdrop element.

Nav updated across all 7 pages. Workspaces is the 7th tab.
Dashboard · Walkthrough · Architecture · Spec · Onboard · Alerts · Workspaces.
 2026-04-20 18:36:51 -05:00
root 6287558493 Push/daemon presence: background digest + /alerts settings page 
Converts the app from 'dashboard you visit' to 'system that finds you.'
Critical for the phone-first staffing shop that won't open a URL —
the system reaches out when something matters.

Daemon:
- Starts once per Bun process (guarded via globalThis sentinel)
- Default interval 15 min (configurable, min 1, max 1440)
- On each cycle, buildDigest() compares current state against prior
  snapshot persisted in mcp-server/data/notification_state.json
- Events detected:
  - risk_escalation: role moved to tight or critical (was ok/watch)
  - deadline_approaching: staffing window falls within warn window
    (default 7 days) AND deadline date differs from prior
  - memory_growth: playbook_memory entries grew by >= 5 since last run

Channels (all opt-out individually via config):
- console: always on, logged to journalctl -u lakehouse-agent
- file: always on, appends JSONL to mcp-server/data/notifications.jsonl
- webhook: optional, POSTs {text, digest} to configured URL
  (Slack incoming-webhook / Discord webhook / any custom endpoint)

Digest format (human-readable, fits in a Slack message):
  LAKEHOUSE DIGEST — 2026-04-20 23:24
  3 staffing deadlines within window:
    • Production Worker — 2d to 2026-04-23 · demand 724
    • Maintenance Tech — 4d to 2026-04-25 · demand 32
    • Electrician — 5d to 2026-04-26 · demand 34
  +779 new playbooks (total 779, 2204 endorsed names)
  snapshot: 0 critical · 0 tight · $275,599,326 pipeline

/alerts page:
- Current status table (daemon state, interval, webhook, last run)
- Config form: enable toggle, interval, deadline warn window, webhook
  URL + label (saved to data/notification_config.json)
- 'Fire a test digest now' button — force a cycle without waiting
- Recent digests panel shows the last 10 dispatches with full text

End-to-end verified live:
- Daemon armed successfully on startup
- First-run digest dispatched to console + file in <1s
- Events detected correctly: 3 deadlines within 7 days from real
  Chicago permit data; 779 playbook entries surfaced as memory growth
- Digest text format is Slack-pastable
- Dispatch records appear in /alerts recent list

TDZ caveat: startAlertsDaemon() invocation moved to end of module so
all const/let in the alerts block evaluate before daemon reads them.
Previously failed with 'Cannot access X before initialization' when
the call lived near the top of the file. Nav added to all 6 pages:
Dashboard · Walkthrough · Architecture · Spec · Onboard · Alerts.
 2026-04-20 18:24:48 -05:00
root 23eb04a145 Onboarding wizard — ingest any staffing CSV in 3 steps 
New /onboard page. Client-facing wizard for getting real data into
the system without engineering help.

Flow:
1. Drop a CSV (or click 'Use the sample as my data' — ships a 25-row
   realistic staffing roster under /samples/staffing_roster_sample.csv)
2. Browser parses client-side. Columns auto-typed (text/int/decimal/
   date). PII flagged by name hint AND content regex (emails, phones).
   First rows previewed. Read-only — nothing written yet.
3. Name the dataset (lowercase+underscores). Commit.
4. Post-commit: dataset is live. Shows 4 next steps the operator can
   take (SQL query, vector index, dashboard search, playbook training).

Backend:
- /onboard serves onboard.html
- /samples/*.csv serves CSV files from mcp-server/samples/ with
  filename validation (only [a-zA-Z0-9_-.]+.csv, prevents path traversal)
- /onboard/ingest forwards multipart/form-data to gateway /ingest/file
  preserving the boundary. The generic /api/* passthrough breaks
  multipart because it reads as text and forwards as JSON; this route
  uses arrayBuffer + original Content-Type.

Verified end-to-end: upload sample roster (25 rows, 12 columns) →
parse in browser → show columns + PII flags + preview → commit →
gateway writes Parquet, registers in catalog → immediately queryable:
  SELECT * FROM onboard_demo2 LIMIT 3
  → Sarah Johnson, Forklift Operator, Chicago, IL, 0.92
Round-trip <1 second.

Nav updated on all pages to link Onboard. Shipped with a sample CSV
so the full flow is demonstrable without real client data.

When a real client shows up, same path — they upload their CSV.
No engineering ticket, no code change, no schema pre-definition.

Security: sample filename regex prevents path traversal. CSV parse
is client-side pure JS (no DOM injection). Commit uses existing
/ingest/file validation (schema fingerprint, PII server-side,
content-hash dedup).
 2026-04-20 18:13:56 -05:00
root 468798c9ac /spec: technical specification — 11-chapter README-equivalent 
J's ask: explain the full architecture so someone reading a README
can dispute it or recreate it. The repo isn't public yet; this page
IS the spec until it is.

Ch1 Repository layout — 13 crates + tests/multi-agent + docs + data,
    with owned responsibility and file path per crate.

Ch2 Data ingest pipeline (8 steps) — sources (file/inbox/DB/cron),
    parse+normalize with ADR-010 conservative typing, PII auto-tag,
    dedup, Parquet write, catalog register with fingerprint gate,
    mark embeddings stale, queryable immediately.

Ch3 Measurement & indexing — row count / fingerprint / owner /
    sensitivity / freshness / lineage per dataset. HNSW vs Lance
    tradeoff table with measured numbers (ADR-019). Autotune loop.
    Per-profile scoping (Phase 17).

Ch4 Contract inference from external signal — Chicago permit feed
    → role mapping → worker count heuristic → timeline → hybrid
    search with boost → pattern discovery → rendered card. All
    pre-computed before staffer opens UI.

Ch5 What a CRM can't do — 11-row comparison table of capabilities.

Ch6 How it gets better over time — three paths:
    - Phase 19 playbook boost (full math)
    - Pattern discovery meta-index
    - Autotune agent

Ch7 Scale story: 20 staffers, 300 contracts, midday +20/+1M surge
    - Async gateway + per-staffer profile isolation + client blacklists
    - 7-step surge handling flow (ingest, stale-mark, incremental refresh,
      degradation, hot-swap, autotune re-enter)
    - Known pain points: Ollama inference serial, RAM ceiling ~5M on
      HNSW (mitigated by Lance), VRAM 1-2 models sequential,
      playbook_memory unbounded.

Ch8 Error surfaces & recovery — 10-row table covering ingest schema
    conflicts, bucket failures, ghost names, dual-agent drift,
    empty searches, Ollama down, gateway restart, schema fingerprint
    divergence. Every failure has a named surface and recovery path.

Ch9 Per-staffer context — active profile, workspace, client blacklist,
    audit trail, daily summary. How 20 staffers don't see the same UI.

Ch10 Day in the life — 07:00 housekeeping → 07:30 refresh → 08:00
     staffer opens → 08:15 drill down → 08:30 Call click → 09:00
     second staffer shares memory → 12:30 surge → 14:00 no-show →
     15:00 new embeddings live → 17:00 retrospective → 22:00
     overnight trials.

Ch11 Known limits & non-goals — deferred (rate/margin, push, confidence
     calibration, neural re-ranker, pm compaction, call_log cross-ref)
     and explicitly out-of-scope (cloud, ACID, streaming, CRM replace,
     proprietary formats, hard multi-tenant).

Also: nav updated on /dashboard, /console, /proof to link /spec.
Every architectural claim in the spec cites either a code path, an
ADR number, or a phase reference so someone skeptical can target
the specific artifact.
 2026-04-20 17:56:18 -05:00
root 76bfa2c8d7 /proof: explain the dual-agent recursive architecture with citations 
Previous page was numeric claims without explanations — 'sub-100ms SQL',
'500K vectors in 341ms' etc. Accurate but undefendable without math,
code paths, and ADR references. Expanded to 8 chapters:

Ch1 — Live receipts (unchanged: real gateway tests, pass/fail, timing)

Ch2 — Architecture. 13-crate diagram with per-crate responsibility
      table and file paths. gateway → catalogd/queryd/vectord/ingestd
      + aibridge → object_store. References ADRs 1-20.

Ch3 — Dual-agent recursive consensus loop (NEW)
      - Role specialization (executor=optimist, reviewer=pessimist)
      - Parallel orchestration via Promise.all
      - Recursive: sealed playbooks feed playbook_memory → next query
      - Termination math: sealed | tool-error abort | drift abort |
        turn-cap abort — every path dumps forensic log
      - File refs: tests/multi-agent/agent.ts, orchestrator.ts,
        scenario.ts, run_e2e_rated.ts

Ch4 — Playbook memory feedback loop (NEW)
      - PlaybookEntry shape with embedding
      - Full boost math: similarity * base_weight * decay * penalty
        / n_workers, capped at MAX_BOOST_PER_WORKER
      - Temporal decay (e^-age/30, 30d half-life)
      - Negative signal (0.5^failures)
      - Why k=200: narrow cosine discrimination in nomic-embed-text
      - Evidence: compounding test 0 → 0.250 cap in 3 seeds
      - persist_sql write-through
      - Pattern discovery (Path 2 meta-index)
      - File: crates/vectord/src/playbook_memory.rs

Ch5 — ADR citations for each key choice
      ADR-001, 008, 012, 015, 019, 020 + Phase 19 design note

Ch6 — Live scale data (unchanged: pulled from /proof.json)

Ch7 — Reproduction recipes: curl for health, sql, hybrid with boost,
      patterns, pm stats, and the full dual-agent scenario run

Ch8 — Honest limits (unchanged: synthetic workers_500k, 1K candidates
      misaligned to call_log, 7B model imperfection, no rate/margin)

Every architectural claim now cites either the code path
(crates/.../src/file.rs::fn_name) or the ADR (docs/DECISIONS.md).
Someone disputing the system has specific targets to attack.

Mechanism unchanged: /proof serves mcp-server/proof.html via
Bun.file. /proof.json still returns the live test data the page
consumes client-side.
 2026-04-20 17:49:08 -05:00
root 05f2e42c45 Rebuild /console as narrative walkthrough for a skeptical staffer 
Old console was a chat playground. New console is a guided,
chapter-based explanation that a non-technical staffing staffer
can read top-down and finish convinced — without needing to
understand any of the underlying technology.

Six chapters, each loading live data:

1. Right now, this system is already thinking
   Four stats cards pulled live: construction pipeline $, predicted
   worker demand, rows under management, playbooks remembered. Then
   a narrative that names the current alert posture (critical/tight/ok).

2. The demand signal is real, not made up
   Expandable rows per Chicago permit work_type, with a direct link to
   data.cityofchicago.org for verification. Pill labeled LIVE ·
   DATA.CITYOFCHICAGO.ORG leaves no ambiguity.

3. Where your own data would live
   Catalog enumerated with three pill classes:
   - SWAP FOR YOUR DATA (purple) — the synthetic tables that would
     be replaced by the client's ATS/CRM/call-log exports
   - SYSTEM-GENERATED (blue) — playbook memory, threat_intel, kb_*
     produced by the system itself
   Row counts + columns visible. Names it honestly.

4. Watch the system rank candidates in real time
   Takes the freshest Chicago permit, walks the staffer through all
   three steps (derive need → narrow via SQL → rank + boost), shows
   the top-5 workers with why, boost chip, memory chip, timeline,
   and a plain-English narrative of the CRM gap.

5. Every action compounds
   Playbook memory count + sample + narrative about what it means
   when the staffer logs a fill.

6. Try it yourself
   Free-text input hitting /intelligence/chat, renders response
   with memory chip + boost chips + ranked workers.

Security: all API-derived strings go through textContent or
el(tag,cls,text) helper. Zero innerHTML usage on dynamic content.
Passes security reminder hook.

File size: 419 → ~500 lines. Visual style matches the dashboard
(same palette, typography, chip styles) so the two pages feel
like one app.
 2026-04-20 17:35:45 -05:00
root bb1b471c67 Predictive staffing forecast + per-contract timeline 
J's ask: move the system from retrospective ranking to predictive
anticipation. Show it tracks the clock, not just the roster.

New endpoint /intelligence/staffing_forecast:
- Pulls 30-day Chicago permit window (200 permits)
- Maps work_type → role via industry heuristic
- Aggregates predicted worker demand per role
- Joins IL bench supply (workers_500k state='IL' group by role)
- Computes coverage_pct, reliable_coverage_pct
- Classifies risk: critical/tight/watch/ok
- Computes earliest staffing deadline per role
  (permit issue_date + 31d = 45d construction start - 14d window)
- Surfaces recent Chicago playbook ops for the role-specific memory

New UI 'Staffing Forecast' section ABOVE Live Contracts:
- Top card: total construction value, permit count, workers needed,
  critical/tight role count
- Per-role rows: demand vs available supply, coverage %, deadline
  with red/amber/green urgency coloring

Per-contract timeline on Live Contracts:
- estimated_construction_start, staffing_window_opens, days_to_deadline
- urgency classification: overdue/urgent/soon/scheduled
- card border colored by urgency
- timeline line explicitly shows recruiter: OVERDUE/URGENT + days count

This is the 'system already thinks about when, not just who' surface
J was asking for. CRMs store; this anticipates.
 2026-04-20 17:24:17 -05:00
root 2595d48535 Gap fixes: pattern fallback, narrative citations, call_log plumbing 
Closing trust-breaks surfaced in the strategic audit.

A — MEMORY chip renders even when sparse:
Previously rendered nothing when no trait crossed threshold, which
recruiters would read as "system has no signal." Now explicitly
says "memory is sparse for this role+geo — no trait crossed
threshold" or "no similar past playbooks yet — first fill of this
kind will seed it." Honest when it doesn't know.

B — Removed /intelligence/learn dead endpoint:
Legacy CSV-writer path that destructively re-wrote
successful_playbooks. /log and /log_failure replace it cleanly.
Leaving dead code confuses future maintainers.

C — Narrative tooltips on Endorsed chips:
Hovering the green "Endorsed · N playbooks" chip now fetches
the worker's past operations from successful_playbooks_live and
shows a story: "Maria — past endorsements: • Welder x2 in
Toledo (2026-04-15), • Welder x1 in Toledo (2026-04-18)..."
Falls back to honest "narrative unavailable" if the seed
didn't land in SQL.

D — call_log infrastructure in worker modal:
New "Recent Contact" section queries call_log JOIN candidates by
name. Surfaces last 3 call entries with timestamp, recruiter,
disposition, duration. When empty (which is today's reality —
candidates table only has 1000 rows vs call_log's higher IDs),
shows an honest message about the data gap and what real ATS
integration would unlock.

Honest call: D ships infrastructure. Actual utility depends on
aligning candidate IDs between the candidates table and
call_log — current synthetic data doesn't cross-ref cleanly.
When real ATS data lands, this section becomes the
"system knows who we called yesterday" feature the recruiter
needs.

Deferred (would require a dedicated session):
- Rate awareness (needs worker pay_rate + contract bill_rate)
- Push / background daemon (Slack/SMS/email integration)
- Confidence calibration (needs a probabilistic ranking layer)
 2026-04-20 17:20:22 -05:00
root 1f56630d5d #3: Worker profile modal shows past playbook history 
Click any worker card → modal now includes a 'Past Playbooks' section
that queries successful_playbooks_live for any row where this worker's
name appears in the result field. Shows up to 8 most recent with
operation, timestamp, approach, and context.

When empty: 'No prior playbooks for NAME yet. First placement builds
the first entry.' — makes the institutional-memory claim visible to
the recruiter: the system is tracking everyone, not just the ones
that sealed this session.

Also added Call / SMS / No-show buttons to the modal action row
(matching the card-level buttons from #1). Every worker-card path
now trains the system.

Closes the user-visible side of Phase 19 — patterns surface during
search (Pass A), boosts fire in ranking (Phase 19 core), and now
the worker's own profile shows the full history that informs those
boosts. Institutional memory legibility, per J's ask.
 2026-04-20 16:21:27 -05:00
root cdd12a1438 #2: Per-client worker blacklists 
New endpoints:
- POST   /clients/:client/blacklist            { worker_id, name?, reason? }
- GET    /clients/:client/blacklist            → { client, entries }
- DELETE /clients/:client/blacklist/:worker_id → { removed, total }

Bun /search accepts optional `client` field. When present, loads that
client's blacklist and appends `AND worker_id NOT IN (...)` to the
SQL filter. Zero-cost if unused; clean trust-break avoidance when a
client has previously flagged a worker.

Persistence: mcp-server/data/client_blacklists.json, synchronous
writes via Bun.write. Scale target is hundreds of entries per client
tops — JSON is fine until we hit 10K+ per client.

Verified: worker_id 9326 (Carmen Green) blacklisted for AcmeCorp,
same Chicago Electrician search with client=AcmeCorp returns 196
sql_matches vs 197 without — exactly one excluded.
 2026-04-20 16:20:17 -05:00
root 4aea71d213 #1: Close recruiter feedback loop — Call/SMS/No-show fire /log and /log_failure 
Every worker-card button in the dashboard now trains the Phase 19
system directly:

- Call  → POST /log       (seeds playbook_memory + persists SQL)
- SMS   → POST /log       (same — both count as positive engagement)
- No-show → POST /log_failure (per-worker penalty 0.5^n on future boost)

Buttons flash status (Logged / Flagged / Ghost) for 1.4s on success,
then re-enable. Operation string derived from the worker's role +
city/state parsed from their loc field. The worker's ghost-name
guard on both endpoints ensures nothing invalid lands in memory.

Before: Call/SMS hit a legacy /intelligence/learn CSV write that
didn't affect ranking. No failure capture existed.

Now: recruiter using the app IS the training signal. Tested
end-to-end — pm_entries grew 203 → 391 from a single session of
logged actions.
 2026-04-20 16:19:14 -05:00
root 72ee8f006f k=200 on /search and /match too — consistency with compounding default 2026-04-20 15:41:39 -05:00
root 99ab0fe623 A+B: patterns in main search + compounding bump 
A — Patterns surface in main Worker Search:
  /intelligence/chat smart_search fallback now calls /patterns in
  parallel with hybrid, returns discovered_pattern + matched count.
  search.html doSearch renders a green "MEMORY (N playbooks): ..."
  chip above results so every recruiter query shows the meta-index
  dimension, not just live-contract cards.

B — Compounding proven and default-k bumped:
  Direct compounding test on Chicago Electrician:
  - Run 0 (no seeds): Carmen Green not in top-5, boost 0
  - After 3 seeds of identical operation: boost +0.250 (capped),
    3 citations, lifted to #1. Each seed adds 1 citation. Cap
    prevents one worker from dominating future searches.
  - Required k=200 (not 25 or 50) — embedding band is narrow
    (cosines 0.55-0.67 across all playbooks regardless of geo).
  - Bumped defaults on /search, permit_contracts, and smart_search
    to playbook_memory_k=200. Brute-force sub-ms at this scale.
 2026-04-20 15:41:12 -05:00
root 5c39c74fe4 Live Contracts canvas: Chicago permits × workers_500k × playbook patterns 
New devop.live/lakehouse section pairs live public Chicago building
permits with derived staffing contracts, ranked candidates from the
500K worker bench, and meta-index discovered patterns per role+geo.
Makes the Phase 19 boost + Path 2 pattern discovery visible on real
external data, without needing a paying client to demo.

Backend:
- New /intelligence/permit_contracts endpoint
- Fetches 6 recent Chicago permits > $250K from the Socrata API
- Derives proposed fill: 1 worker per $150K of permit value (capped 2-8)
- For each: /vectors/hybrid with use_playbook_memory=true,
  playbook_memory_k=25, auto availability>0.5 filter
- For each: /vectors/playbook_memory/patterns with k=25 min_freq=0.3
- Returns permit + proposed contract + top 5 candidates with boosts
  and citations + discovered pattern + pattern_matched count

Frontend:
- New "Live Contracts" section on search.html between today's sim
  contracts and Market Intelligence
- Per-permit card: cost + work_type + address + proposed role/count
  + pool size + top 3 candidates (with endorsement chip when boost
  fires) + memory-derived pattern ("MEMORY (N playbooks): recurring
  certifications: OSHA-10 47%, Forklift... · archetype mostly: ...")

Real working demo even without paying clients: shows the system
operating on genuinely external data with our synthetic-data-derived
learning applied.
 2026-04-20 15:36:14 -05:00
root f8e8d25b5f Unblock complex scenarios: JSON tolerance + optional question + mistral exec 
parseAction now strips stray `)` before `}` and trailing commas —
qwen2.5 emits those regularly on tool_call outputs; soft-fix beats
retry-loops. hybrid_search no longer hard-requires `question`; defaults
to "qualified available workers" when the model drops it (mistral's
most common failure mode on complex events).

Kept original TOOL_CATALOG shape (args examples only, not full
action envelopes). The verbose few-shot version from the prior
iteration confused mistral into wrapping propose_done as tool_call.

Scenario V7 result: expansion (5 Forklift Ops) and emergency
(4 Loaders) — previously-failing complex events — now seal reliably.
Pool sizes: 687 and 380 from 500K corpus. Patterns endpoint produces
real operator-actionable signals:
  expansion: "recurring certifications: Forklift (40%), OSHA-10 (40%)
             · recurring skills: mill (40%) · archetype mostly: leader
             · reliability median 0.83"
Baseline + recurring are now flaky (inverted trade-off, pure
model-reliability variance).
 2026-04-20 15:28:30 -05:00
root 1274ab2cb3 Scenario harness: Path 1+2 integration + schema hardening 
Upgrades to tests/multi-agent/scenario.ts to exercise the full Path 1+2
feature set on a real warehouse-client week (5 events on one client):

- Hard SCHEMA ENFORCEMENT block in every event's guidance. Prior runs
  had mistral read narrative words ("shift", "recurring", "expansion")
  as SQL column names. Schema is now locked explicitly with valid
  columns listed and CAST guidance for availability + reliability.
- playbook_memory_k bumped 10 → 100 to match server default.
- Canonical short seed text (operation + "{kind} fill via hybrid
  search" + "{role} fill in {city}, {state}"). Verbose LLM rationales
  dilute embeddings and silently kill boost (Pass 1 finding).
- /vectors/playbook_memory/mark_failed fires automatically on
  misplacement events — records the no-shower's failure so future
  searches for same city+role dampen their boost.
- /vectors/playbook_memory/patterns call per event — surfaces what the
  meta-index discovered (recurring certs/skills/archetype/reliability)
  for that query into the dispatch log and retrospective.
- Retrospective now includes a workers-touched audit table (every
  worker who reached a decision, with outcome column) and a
  discovered-patterns-evolution section across events.

Honest limitations this surfaced in the real run:
- mistral's executor prompt-adherence degrades on high-count events
  (5+ fills) and scenario-specific language (emergency/misplacement).
  3 of 5 events aborted via drift guard. Baseline + recurring sealed
  cleanly with real fills + SMS + emails + seeded playbooks.
- worker_id resolution returns "undefined" for some names when name
  matching is ambiguous in workers_500k (multiple workers with same
  name in same city).
 2026-04-20 15:09:14 -05:00
root 95c26f04f8 Path 1 negative signal + Path 2 pattern discovery + name validation 
New:
- /vectors/playbook_memory/patterns: meta-index pattern discovery.
  Given a query, finds top-K similar playbooks, pulls each endorsed
  worker's full workers_500k profile, aggregates shared traits (cert
  frequencies, skill frequencies, modal archetype, reliability
  distribution), returns a human-readable discovered_pattern. Surfaces
  signals operators didn't explicitly query — the original PRD's
  "identify things we didn't know" dimension.
- /vectors/playbook_memory/mark_failed: records worker failures per
  (city, state, name). compute_boost_for applies 0.5^n penalty per
  recorded failure, so 3 failures quarter a worker's positive boost and
  5 effectively zero it. Path 1 negative signal — recruiter trust
  depends on the system NOT recommending people who no-showed.
- Bun /log_failure: validates failed_names against workers_500k
  (same ghost-guard as /log), forwards to /mark_failed.

Improved:
- /log now validates endorsed_names against workers_500k for the
  contract's city+state before seeding. Ghost names (names that don't
  correspond to real workers) are rejected in the response and excluded
  from the seed, preventing silent boost failures.
- Bun /search auto-appends `CAST(availability AS DOUBLE) > 0.5` to
  sql_filter when the caller didn't constrain availability. Opt out
  with `include_unavailable: true`. Recruiter trust bug: surfacing
  already-placed workers breaks the first call.
- DEFAULT_TOP_K_PLAYBOOKS 25 → 100. Direct cosine measurement showed
  similarities cluster 0.55-0.67 across all playbooks regardless of
  geo, so k=25 missed relevant geo-matched playbooks. Brute-force is
  still sub-ms at this size.

Verified end-to-end on live data:
- Ghost names rejected on /log + /log_failure
- Availability filter drops unavailable workers from candidate pool
- Pattern discovery on unseen Cleveland OH Welder query returned
  recurring skills (first aid 43%, grinder 43%, blueprint 43%) and
  modal archetype (specialist) across 20 semantically similar past
  playbooks in 0.24s
- Negative signal: Helen Sanchez boost dropped +0.250 → +0.163 after
  3 failures recorded via /log_failure (34% reduction)
 2026-04-20 14:55:46 -05:00
root 20b0289aa9 /log validates endorsed names + /search auto-appends availability>0.5 
Two gap-fills surfaced by the real test on 2026-04-20:

1. /log no longer seeds endorsed_names that don't exist in workers_500k
   for the contract's (city, state). Previously accepted ghost names
   silently (entry count grew, SQL row landed, but boost never fired
   because no real worker chunk matched the stored tuple). Response now
   reports rejected_ghost_names and explains why seeding was skipped.

2. Bun /search auto-appends `CAST(availability AS DOUBLE) > 0.5` to
   sql_filter when the caller didn't constrain availability themselves.
   Recruiters expect "available workers" by default — surfacing someone
   on an active placement would break trust on first contact.
   Opt out with `include_unavailable: true`.

Verified: ghost names rejected end-to-end, real names accepted, mixed
input handled correctly. Availability filter drops ~10 workers from a
305-row Cleveland OH Welder pool to 295 actually-available.
 2026-04-20 14:44:12 -05:00
153 changed files with 19620 additions and 2023 deletions
Expand all files Collapse all files
Cargo.lock
Generated
+109 -7
View File
@@ -1254,7 +1254,7 @@ version = "3.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "519bd3116aeeb42d5372c29d982d16d0170d3d4a5ed85fc7dd91642ffff3c67c" checksum = "519bd3116aeeb42d5372c29d982d16d0170d3d4a5ed85fc7dd91642ffff3c67c"
dependencies = [ dependencies = [
"darling", "darling 0.21.3",
"ident_case", "ident_case",
"prettyplease", "prettyplease",
"proc-macro2", "proc-macro2",
@@ -1759,6 +1759,17 @@ dependencies = [
"cfg-if", "cfg-if",
] ]
[[package]]
name = "croner"
version = "3.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4aa42bcd3d846ebf66e15bd528d1087f75d1c6c1c66ebff626178a106353c576"
dependencies = [
"chrono",
"derive_builder",
"strum 0.27.2",
]
[[package]] [[package]]
name = "crossbeam" name = "crossbeam"
version = "0.8.4" version = "0.8.4"
@@ -1862,14 +1873,38 @@ dependencies = [
"memchr", "memchr",
] ]
[[package]]
name = "darling"
version = "0.20.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc7f46116c46ff9ab3eb1597a45688b6715c6e628b5c133e288e709a29bcb4ee"
dependencies = [
"darling_core 0.20.11",
"darling_macro 0.20.11",
]
[[package]] [[package]]
name = "darling" name = "darling"
version = "0.21.3" version = "0.21.3"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9cdf337090841a411e2a7f3deb9187445851f91b309c0c0a29e05f74a00a48c0" checksum = "9cdf337090841a411e2a7f3deb9187445851f91b309c0c0a29e05f74a00a48c0"
dependencies = [ dependencies = [
"darling_core", "darling_core 0.21.3",
"darling_macro", "darling_macro 0.21.3",
]
[[package]]
name = "darling_core"
version = "0.20.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0d00b9596d185e565c2207a0b01f8bd1a135483d02d9b7b0a54b11da8d53412e"
dependencies = [
"fnv",
"ident_case",
"proc-macro2",
"quote",
"strsim",
"syn 2.0.117",
] ]
[[package]] [[package]]
@@ -1886,13 +1921,24 @@ dependencies = [
"syn 2.0.117", "syn 2.0.117",
] ]
[[package]]
name = "darling_macro"
version = "0.20.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc34b93ccb385b40dc71c6fceac4b2ad23662c7eeb248cf10d529b7e055b6ead"
dependencies = [
"darling_core 0.20.11",
"quote",
"syn 2.0.117",
]
[[package]] [[package]]
name = "darling_macro" name = "darling_macro"
version = "0.21.3" version = "0.21.3"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d38308df82d1080de0afee5d069fa14b0326a88c14f15c5ccda35b4a6c414c81" checksum = "d38308df82d1080de0afee5d069fa14b0326a88c14f15c5ccda35b4a6c414c81"
dependencies = [ dependencies = [
"darling_core", "darling_core 0.21.3",
"quote", "quote",
"syn 2.0.117", "syn 2.0.117",
] ]
@@ -3149,6 +3195,37 @@ dependencies = [
"serde_core", "serde_core",
] ]
[[package]]
name = "derive_builder"
version = "0.20.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "507dfb09ea8b7fa618fcf76e953f4f5e192547945816d5358edffe39f6f94947"
dependencies = [
"derive_builder_macro",
]
[[package]]
name = "derive_builder_core"
version = "0.20.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2d5bcf7b024d6835cfb3d473887cd966994907effbe9227e8c8219824d06c4e8"
dependencies = [
"darling 0.20.11",
"proc-macro2",
"quote",
"syn 2.0.117",
]
[[package]]
name = "derive_builder_macro"
version = "0.20.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ab63b0e2bf4d5928aff72e83a7dace85d7bba5fe12dcc3c5a572d78caffd3f3c"
dependencies = [
"derive_builder_core",
"syn 2.0.117",
]
[[package]] [[package]]
name = "derive_more" name = "derive_more"
version = "2.1.1" version = "2.1.1"
@@ -3713,7 +3790,7 @@ version = "0.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f43e744e4ea338060faee68ed933e46e722fb7f3617e722a5772d7e856d8b3ce" checksum = "f43e744e4ea338060faee68ed933e46e722fb7f3617e722a5772d7e856d8b3ce"
dependencies = [ dependencies = [
"darling", "darling 0.21.3",
"proc-macro2", "proc-macro2",
"quote", "quote",
"syn 2.0.117", "syn 2.0.117",
@@ -4002,6 +4079,7 @@ dependencies = [
"opentelemetry_sdk", "opentelemetry_sdk",
"proto", "proto",
"queryd", "queryd",
"reqwest",
"serde", "serde",
"serde_json", "serde_json",
"shared", "shared",
@@ -4598,6 +4676,7 @@ dependencies = [
"bytes", "bytes",
"catalogd", "catalogd",
"chrono", "chrono",
"croner",
"csv", "csv",
"lopdf", "lopdf",
"mysql_async", "mysql_async",
@@ -5096,7 +5175,7 @@ dependencies = [
"prost-types 0.14.3", "prost-types 0.14.3",
"rand 0.9.2", "rand 0.9.2",
"snafu", "snafu",
"strum", "strum 0.26.3",
"tokio", "tokio",
"tracing", "tracing",
"xxhash-rust", "xxhash-rust",
@@ -7857,7 +7936,16 @@ version = "0.26.3"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8fec0f0aef304996cf250b31b5a10dee7980c85da9d759361292b8bca5a18f06" checksum = "8fec0f0aef304996cf250b31b5a10dee7980c85da9d759361292b8bca5a18f06"
dependencies = [ dependencies = [
"strum_macros", "strum_macros 0.26.4",
]
[[package]]
name = "strum"
version = "0.27.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "af23d6f6c1a224baef9d3f61e287d2761385a5b88fdab4eb4c6f11aeb54c4bcf"
dependencies = [
"strum_macros 0.27.2",
] ]
[[package]] [[package]]
@@ -7873,6 +7961,18 @@ dependencies = [
"syn 2.0.117", "syn 2.0.117",
] ]
[[package]]
name = "strum_macros"
version = "0.27.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7695ce3845ea4b33927c055a39dc438a45b059f7c1b3d91d38d10355fb8cbca7"
dependencies = [
"heck",
"proc-macro2",
"quote",
"syn 2.0.117",
]
[[package]] [[package]]
name = "subprocess" name = "subprocess"
version = "0.2.15" version = "0.2.15"
@@ -8814,12 +8914,14 @@ dependencies = [
"bytes", "bytes",
"catalogd", "catalogd",
"chrono", "chrono",
"futures",
"instant-distance", "instant-distance",
"object_store", "object_store",
"parquet 55.2.0", "parquet 55.2.0",
"queryd", "queryd",
"serde", "serde",
"serde_json", "serde_json",
"sha2",
"shared", "shared",
"storaged", "storaged",
"tokio", "tokio",
auditor/README.md
+53
View File
@@ -0,0 +1,53 @@
# Lakehouse Claim Auditor
A Bun sub-agent that watches open PRs on Gitea, reads the ship-claims
in commit messages and PR bodies, and **hard-blocks** merges when the
code doesn't back the claim.
Rationale: when "compiles + one curl works" gets called "phase shipped,"
placeholder code accumulates. This auditor runs every 90s, fetches
each open PR, and subjects it to four checks:
1. **Static diff** — grep/parse looking for placeholder patterns
2. **Dynamic** — runs the never-before-executed hybrid test fixture
3. **Cloud inference** — asks `gpt-oss:120b` via `/v1/chat` to
identify gaps in the diff
4. **KB query** — looks up `data/_kb/` + observer for prior failure
patterns on similar claims
Verdict is assembled, posted to Gitea as:
- A **failing commit status** (hard block — branch protection
prevents merge)
- A **review comment** explaining every finding
## Run manually
```bash
cd /home/profit/lakehouse
bun run auditor/index.ts
```
Defaults: polls every 90s, stops on `auditor.paused` file present.
## State
- `data/_auditor/state.json` — last-audited head SHA per PR
- `data/_auditor/verdicts/{pr}-{sha}.json` — per-run verdict record
## Where YOU edit
`auditor/policy.ts` — the verdict assembler. Controls which findings
block vs warn vs inform. All other code is mechanical: fetching,
running checks, posting to Gitea.
## Hard-block mechanism
1. Commit status is posted as `failure` with context `lakehouse/auditor`
2. If `main` branch protection requires `lakehouse/auditor` status
to pass, Gitea prevents merge
3. When code is fixed and re-audit passes, status flips to `success`,
merge unblocks
Enable branch protection (one-time, via Gitea UI or API):
- `POST /repos/profit/lakehouse/branch_protections`
- `{"branch_name": "main", "required_status_checks": {"contexts": ["lakehouse/auditor"]}}`
auditor/audit.ts
+171
View File
@@ -0,0 +1,171 @@
// Orchestrator — runs all four checks on a PR, assembles a verdict,
// posts to Gitea. This is task #8's integration layer; the poller
// (task #9) calls this once per PR on every fresh head SHA.
//
// Hard-block mechanism: commit status posted with state="failure"
// and context="lakehouse/auditor". If `main` branch protection
// requires that context to pass, merge is physically impossible
// until the auditor re-audits a fixed commit and flips the status
// to "success".
//
// Human-readable reasoning: posted as a PR issue comment (not a
// review — reviews have self-review restrictions on Gitea and the
// auditor currently uses the same PAT as the PR author).
import { readFile, writeFile, mkdir } from "node:fs/promises";
import { join } from "node:path";
import type { PrSnapshot, Verdict, Finding } from "./types.ts";
import { getPrDiff, postCommitStatus, postIssueComment } from "./gitea.ts";
import { parseClaims } from "./claim_parser.ts";
import { assembleVerdict } from "./policy.ts";
import { runStaticCheck } from "./checks/static.ts";
import { runDynamicCheck } from "./checks/dynamic.ts";
import { runInferenceCheck } from "./checks/inference.ts";
import { runKbCheck } from "./checks/kb_query.ts";
const VERDICTS_DIR = "/home/profit/lakehouse/data/_auditor/verdicts";
export interface AuditOptions {
// Skip the cloud inference call (fast path for iteration). Default false.
skip_inference?: boolean;
// Skip the dynamic check (avoid running the hybrid fixture every PR,
// since it hits live services and mutates playbook state). Default false
// on `main`-branch-target PRs, true when auditing feature branches
// where the fixture would pollute state. Caller decides.
skip_dynamic?: boolean;
// Skip Gitea posting — useful for dry-runs / local testing.
// Default false.
dry_run?: boolean;
}
export async function auditPr(pr: PrSnapshot, opts: AuditOptions = {}): Promise<Verdict> {
const t0 = Date.now();
const diff = await getPrDiff(pr.number);
const { claims } = parseClaims(pr);
// Run checks in parallel where they don't share mutable state.
// Static + kb_query + inference are all read-only. Dynamic mutates
// playbook state (nonce-scoped per run, but still live) so if
// skip_dynamic is false we still run it in parallel — the mutation
// is namespaced.
const [staticFindings, dynamicFindings, inferenceFindings, kbFindings] = await Promise.all([
runStaticCheck(diff),
opts.skip_dynamic ? Promise.resolve(stubFinding("dynamic", "skipped by options")) : runDynamicCheck(),
opts.skip_inference ? Promise.resolve(stubFinding("inference", "skipped by options")) : runInferenceCheck(claims, diff),
runKbCheck(claims),
]);
const allFindings: Finding[] = [
...staticFindings,
...dynamicFindings,
...inferenceFindings,
...kbFindings,
];
const duration_ms = Date.now() - t0;
const metrics = {
audit_duration_ms: duration_ms,
findings_total: allFindings.length,
findings_block: allFindings.filter(f => f.severity === "block").length,
findings_warn: allFindings.filter(f => f.severity === "warn").length,
findings_info: allFindings.filter(f => f.severity === "info").length,
claims_strong: claims.filter(c => c.strength === "strong").length,
claims_moderate: claims.filter(c => c.strength === "moderate").length,
claims_weak: claims.filter(c => c.strength === "weak").length,
claims_total: claims.length,
diff_bytes: diff.length,
};
const verdict = assembleVerdict(allFindings, metrics, pr.number, pr.head_sha);
await persistVerdict(verdict);
if (!opts.dry_run) {
await postToGitea(verdict);
}
return verdict;
}
async function persistVerdict(v: Verdict): Promise<void> {
await mkdir(VERDICTS_DIR, { recursive: true });
const filename = `${v.pr_number}-${v.head_sha.slice(0, 12)}.json`;
await writeFile(join(VERDICTS_DIR, filename), JSON.stringify(v, null, 2));
}
export async function postToGitea(v: Verdict): Promise<void> {
// 1. Commit status — the hard block signal (if branch protection
// is configured to require lakehouse/auditor on main).
const state = v.overall === "approve" ? "success" : "failure";
await postCommitStatus({
sha: v.head_sha,
state,
context: "lakehouse/auditor",
description: v.one_liner,
target_url: "", // no URL yet; could point to a verdicts dashboard
});
// 2. Issue comment — the reasoning. Gated so we don't spam the PR
// with identical comments on re-audits of the same SHA. Caller
// (poller) ensures we only re-audit fresh SHAs, but a dedup
// marker inside the body keeps it idempotent if re-run.
const body = formatReviewBody(v);
await postIssueComment({ pr_number: v.pr_number, body });
}
function formatReviewBody(v: Verdict): string {
const byCheck: Record<string, Finding[]> = {};
for (const f of v.findings) {
(byCheck[f.check] ||= []).push(f);
}
const verdictEmoji =
v.overall === "approve" ? "✅" :
v.overall === "request_changes" ? "⚠️" :
"🛑";
const lines: string[] = [];
lines.push(`## Auditor verdict: ${verdictEmoji} \`${v.overall}\``);
lines.push("");
lines.push(`**One-liner:** ${v.one_liner}`);
lines.push(`**Head SHA:** \`${v.head_sha.slice(0, 12)}\``);
lines.push(`**Audited at:** ${v.audited_at}`);
lines.push("");
// Per-check sections, only if the check produced findings.
const checkOrder = ["static", "dynamic", "inference", "kb_query"] as const;
for (const check of checkOrder) {
const fs = byCheck[check] ?? [];
if (fs.length === 0) continue;
const bySev = {
block: fs.filter(f => f.severity === "block").length,
warn: fs.filter(f => f.severity === "warn").length,
info: fs.filter(f => f.severity === "info").length,
};
lines.push(`<details><summary><b>${check}</b> — ${fs.length} findings (${bySev.block} block, ${bySev.warn} warn, ${bySev.info} info)</summary>`);
lines.push("");
for (const f of fs) {
const mark = f.severity === "block" ? "🛑" : f.severity === "warn" ? "⚠️" : "ℹ️";
lines.push(`${mark} **${f.severity}** — ${f.summary}`);
for (const e of f.evidence.slice(0, 3)) {
lines.push(` - \`${e.slice(0, 180).replace(/\n/g, " ")}\``);
}
}
lines.push("");
lines.push("</details>");
lines.push("");
}
lines.push("### Metrics");
lines.push("```json");
lines.push(JSON.stringify(v.metrics, null, 2));
lines.push("```");
lines.push("");
lines.push(`<sub>Lakehouse auditor · SHA ${v.head_sha.slice(0, 8)} · re-audit on new commit flips the status automatically.</sub>`);
return lines.join("\n");
}
function stubFinding(check: "dynamic" | "inference", why: string): Finding[] {
return [{ check, severity: "info", summary: `${check} check skipped — ${why}`, evidence: [why] }];
}
auditor/checks/dynamic.ts
+91
View File
@@ -0,0 +1,91 @@
// Dynamic execution check — runs the hybrid fixture and maps its
// layer results to auditor Findings.
//
// A layer that fails with a "not implemented / 404 / slice N" error
// gets severity=info (honest placeholder signal). A layer that fails
// any other way gets severity=warn (something actually broke).
// An info-level summary finding is always emitted carrying the real
// numbers — shipped/placeholder phase counts, per-layer latency.
import { runHybridFixture } from "../fixtures/hybrid_38_40_45.ts";
import type { Finding } from "../types.ts";
const PLACEHOLDER_MARKERS = [
"unimplemented",
" 404 ", "(404)", " 405 ", "(405)",
"slice 3", "slice 4", "slice 5",
"endpoint not built", "not yet",
];
function isPlaceholderFailure(err?: string): boolean {
if (!err) return false;
const low = err.toLowerCase();
return PLACEHOLDER_MARKERS.some(m => low.includes(m.toLowerCase()));
}
export async function runDynamicCheck(): Promise<Finding[]> {
const findings: Finding[] = [];
let result;
try {
result = await runHybridFixture();
} catch (e) {
// Fixture itself crashed — can't run dynamic check at all.
return [
{
check: "dynamic",
severity: "warn",
summary: `hybrid fixture crashed before completing: ${(e as Error).message.slice(0, 140)}`,
evidence: [(e as Error).message],
},
];
}
// Per-layer findings for every non-ok layer.
for (const layer of result.layers) {
if (layer.ok) continue;
const placeholder = isPlaceholderFailure(layer.error);
findings.push({
check: "dynamic",
severity: placeholder ? "info" : "warn",
summary: placeholder
? `hybrid fixture layer ${layer.layer} (Phase ${layer.phase}) honestly reports unimplemented`
: `hybrid fixture layer ${layer.layer} (Phase ${layer.phase}) failed — not a placeholder, a real failure`,
evidence: [
`evidence: ${layer.evidence.slice(0, 160)}`,
...(layer.error ? [`error: ${layer.error.slice(0, 160)}`] : []),
`latency_ms: ${layer.latency_ms}`,
],
});
}
// One overall summary with real numbers so the report shows what
// DID pass plus per-layer timing.
const metrics_preview = Object.entries(result.real_numbers)
.slice(0, 10)
.map(([k, v]) => `${k}=${v}`);
findings.push({
check: "dynamic",
severity: "info",
summary: `hybrid fixture overall=${result.overall}, shipped [${result.shipped_phases.join(", ")}], placeholder [${result.placeholder_phases.join(", ")}]`,
evidence: metrics_preview.length > 0 ? metrics_preview : ["no metrics emitted"],
});
// If the fixture ran at all but nothing passed, elevate one of the
// summary findings to warn — something more than "all honest
// placeholders" is wrong.
if (result.overall === "fail") {
findings.push({
check: "dynamic",
severity: "warn",
summary: `hybrid fixture: 0 layers passed (overall=fail)`,
evidence: [
"a total fixture fail usually means a precondition service is down",
"(gateway /health / sidecar / Langfuse /v1/chat) — NOT necessarily",
"the PR's code problem. Check service status before blaming the PR.",
],
});
}
return findings;
}
auditor/checks/inference.ts
+206
View File
@@ -0,0 +1,206 @@
// Cloud inference check — wraps the proven run_codereview pattern
// from llm_team_ui.py (same 3-stage framing, same cloud model) to
// critique a PR's claims against its diff.
//
// Proved out 2026-04-22 via /tmp/codereview_runner.py — gpt-oss:120b
// caught a real ternary bug in auditor/fixtures/hybrid_38_40_45.ts
// that unit tests missed. This module reuses the reviewer prompt
// shape (bugs / security / performance / style / edge cases) and
// adds claim-vs-diff specific framing.
//
// Call surface: runInferenceCheck(claims, diff) → Finding[].
// Cloud latency budget: ~60s (gpt-oss:120b reviewer typically 35-50s
// with a 15KB diff + claim list).
import type { Claim, Finding } from "../types.ts";
const GATEWAY = process.env.LH_GATEWAY_URL ?? "http://localhost:3100";
const MODEL = process.env.LH_AUDITOR_REVIEW_MODEL ?? "gpt-oss:120b";
// 40KB comfortably fits gpt-oss:120b's context. PR #1 (~39KB) was
// previously truncated at 15KB causing the reviewer to miss later
// files (gitea.ts, policy.ts) and flag "no Gitea client present" as a
// block finding when the file was simply outside the truncation window.
const MAX_DIFF_CHARS = 40000;
const CALL_TIMEOUT_MS = 120_000;
export async function runInferenceCheck(claims: Claim[], diff: string): Promise<Finding[]> {
if (claims.length === 0) {
return [{
check: "inference",
severity: "info",
summary: "no ship-claims extracted — skipping cloud inference",
evidence: ["parser returned empty claim list; nothing to verify against cloud"],
}];
}
const truncated = diff.length > MAX_DIFF_CHARS
? diff.slice(0, MAX_DIFF_CHARS) + `\n...[${diff.length - MAX_DIFF_CHARS} more chars truncated]`
: diff;
// Build the reviewer prompt in the same shape as run_codereview's
// review stage (llm_team_ui.py:10950), adapted for claim verification:
// "Task: ..."
// "Code: ..."
// "Review: bugs/security/perf/style/edge. Provide corrected code."
// We add: claim list upfront + ask for structured JSON verdict.
const systemMsg = [
"You review pull-request diffs against the author's own ship-claims.",
"For each claim, decide: is it backed by actual code in the diff, or is",
"it placeholder / aspirational / unwired?",
"",
"A claim is BACKED when the diff contains a real code path that delivers",
"the claimed behavior. A claim is NOT BACKED when:",
" - the claim asserts functionality but the diff only adds types/fields",
" with no consumer",
" - the claim mentions tests but no test function was added",
" - the claim claims integration but the integration point is a stub",
" - the diff contains unimplemented!() / todo!() / TODO comments",
" - the claim says 'works end-to-end' but the diff has no end-to-end test",
"",
"Respond with strict JSON only. No prose before or after. Shape:",
"{",
' "claim_verdicts": [',
' {"claim_idx": 0, "backed": false, "evidence": "short reason"}',
" ],",
' "unflagged_gaps": [',
' {"location": "file:line", "summary": "short description"}',
" ]",
"}",
].join("\n");
const userMsg = [
`Ship-claims the author made (numbered 0..N-1):`,
claims.map((c, i) => ` ${i}. [${c.strength}] "${c.text}" at ${c.location}`).join("\n"),
"",
`Diff:`,
"```",
truncated,
"```",
"",
`For each numbered claim above, emit a claim_verdicts entry. For gaps the`,
`author DIDN'T claim but that look like placeholder code, emit unflagged_gaps.`,
`Strict JSON only, matching the shape described. No prose outside JSON.`,
].join("\n");
let resp: Response;
try {
resp = await fetch(`${GATEWAY}/v1/chat`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
provider: "ollama_cloud",
model: MODEL,
messages: [
{ role: "system", content: systemMsg },
{ role: "user", content: userMsg },
],
max_tokens: 3000,
temperature: 0.2,
think: true, // T3 overseer should reason — JSON shape is still required
}),
signal: AbortSignal.timeout(CALL_TIMEOUT_MS),
});
} catch (e) {
// Cloud unreachable → soft-fail. Don't block a PR because the
// reviewer model is down. Static + dynamic + kb still run.
return [{
check: "inference",
severity: "info",
summary: "cloud inference unreachable — skipped",
evidence: [`fetch failed: ${(e as Error).message.slice(0, 180)}`],
}];
}
if (!resp.ok) {
return [{
check: "inference",
severity: "info",
summary: `cloud inference returned ${resp.status} — skipped`,
evidence: [`body: ${(await resp.text()).slice(0, 200)}`],
}];
}
const body: any = await resp.json();
const content: string = body?.choices?.[0]?.message?.content ?? "";
const usage = body?.usage ?? {};
const parsed = extractJson(content);
if (!parsed) {
return [{
check: "inference",
severity: "info",
summary: "cloud returned unparseable output — skipped",
evidence: [
`head: ${content.slice(0, 200)}`,
`tokens: ${usage.total_tokens ?? "?"}`,
],
}];
}
const findings: Finding[] = [];
// One summary info finding so the verdict layer knows the check ran.
findings.push({
check: "inference",
severity: "info",
summary: `cloud review completed (model=${MODEL}, tokens=${usage.total_tokens ?? "?"})`,
evidence: [
`claim_verdicts: ${parsed.claim_verdicts?.length ?? 0}, unflagged_gaps: ${parsed.unflagged_gaps?.length ?? 0}`,
],
});
for (const v of parsed.claim_verdicts ?? []) {
if (v?.backed === false) {
const idx = typeof v.claim_idx === "number" ? v.claim_idx : -1;
const claim = claims[idx];
if (!claim) continue;
// Strong+unbacked = BLOCK. That's the whole point of the auditor.
const sev: Finding["severity"] = claim.strength === "strong" ? "block"
: claim.strength === "moderate" ? "warn"
: "info";
findings.push({
check: "inference",
severity: sev,
claim_text: claim.text,
summary: `cloud: claim not backed — "${claim.text.slice(0, 100)}"`,
evidence: [
`at ${claim.location}`,
`cloud reason: ${String(v.evidence ?? "no reason given").slice(0, 200)}`,
],
});
}
}
for (const g of parsed.unflagged_gaps ?? []) {
findings.push({
check: "inference",
severity: "warn",
summary: `cloud-flagged gap not in any claim: ${String(g?.summary ?? "?").slice(0, 120)}`,
evidence: [`location: ${String(g?.location ?? "?").slice(0, 140)}`],
});
}
return findings;
}
// Lift the first balanced JSON object out of the response. Tolerates
// leading prose, code fences, and model reasoning preamble when the
// cloud model ignored "strict JSON only."
function extractJson(text: string): any | null {
const cleaned = text.replace(/^```(?:json)?\s*/im, "").replace(/```\s*$/im, "");
let depth = 0;
let start = -1;
for (let i = 0; i < cleaned.length; i++) {
const c = cleaned[i];
if (c === "{") {
if (depth === 0) start = i;
depth++;
} else if (c === "}") {
depth--;
if (depth === 0 && start >= 0) {
try { return JSON.parse(cleaned.slice(start, i + 1)); } catch { start = -1; }
}
}
}
return null;
}
auditor/checks/kb_query.ts
+183
View File
@@ -0,0 +1,183 @@
// Local-KB check — reads data/_kb/ + data/_observer/ + data/_bot/
// for prior evidence bearing on this PR's claims. Cheap, offline,
// no model calls. The point: if a claim like "Phase X shipped" has
// a historical record of failing on the same signature before, the
// auditor surfaces that pattern before the cloud check has to
// infer it.
//
// What this check reads (all file-backed, append-only or periodic):
// data/_kb/outcomes.jsonl — per-scenario outcomes (kb.ts)
// data/_kb/error_corrections.jsonl — fail→succeed deltas on same sig
// data/_observer/ops.jsonl — observer ring → disk stream
// data/_bot/cycles/*.json — bot cycle results
//
// Each JSONL line / per-cycle file is small; this check reads tails
// only (last N lines or last M files) to stay cheap on large corpora.
import { readFile, readdir, stat } from "node:fs/promises";
import { join } from "node:path";
import type { Claim, Finding } from "../types.ts";
const KB_DIR = "/home/profit/lakehouse/data/_kb";
const OBSERVER_OPS = "/home/profit/lakehouse/data/_observer/ops.jsonl";
const BOT_CYCLES_DIR = "/home/profit/lakehouse/data/_bot/cycles";
const TAIL_LINES = 500;
const MAX_BOT_CYCLE_FILES = 30;
export async function runKbCheck(claims: Claim[]): Promise<Finding[]> {
const findings: Finding[] = [];
// 1. Recent scenario outcomes: are strong-claim-style phrases showing
// up alongside failed events? That's "we claimed it worked" +
// "it didn't" in the KB.
const scenarioFindings = await checkScenarioOutcomes(claims);
findings.push(...scenarioFindings);
// 2. Error corrections: any of the claims text overlap a
// recently-observed fail→succeed pair? If yes, add context.
const correctionFindings = await checkErrorCorrections(claims);
findings.push(...correctionFindings);
// 3. Bot cycles: any prior bot cycle ended in tests_failed or
// apply_failed on a file this PR is also touching?
const botFindings = await checkBotCycles();
findings.push(...botFindings);
// 4. Observer: count recent error events. High volume = shared
// infra problem, worth flagging (context for other findings).
const obsFindings = await checkObserverStream();
findings.push(...obsFindings);
return findings;
}
async function tailJsonl<T = any>(path: string, n: number): Promise<T[]> {
try {
const raw = await readFile(path, "utf8");
const lines = raw.split("\n").filter(l => l.length > 0);
const slice = lines.slice(-n);
const out: T[] = [];
for (const line of slice) {
try { out.push(JSON.parse(line)); } catch { /* skip malformed */ }
}
return out;
} catch {
return [];
}
}
async function checkScenarioOutcomes(_claims: Claim[]): Promise<Finding[]> {
const outcomes = await tailJsonl<any>(join(KB_DIR, "outcomes.jsonl"), TAIL_LINES);
if (outcomes.length === 0) return [];
const totalEvents = outcomes.reduce((s, o) => s + (o.total_events ?? 0), 0);
const okEvents = outcomes.reduce((s, o) => s + (o.ok_events ?? 0), 0);
const failRate = totalEvents > 0 ? 1 - okEvents / totalEvents : 0;
if (totalEvents === 0) {
return [{
check: "kb_query",
severity: "info",
summary: `KB: no scenario outcomes on file — learning loop is empty`,
evidence: [`data/_kb/outcomes.jsonl has ${outcomes.length} entries with 0 total events`],
}];
}
const recent = outcomes.slice(-10);
const recentFailSigs: string[] = recent
.filter(o => (o.ok_events ?? 0) < (o.total_events ?? 0))
.map(o => o.sig_hash)
.filter(s => typeof s === "string");
const findings: Finding[] = [{
check: "kb_query",
severity: failRate > 0.3 ? "warn" : "info",
summary: `KB: ${outcomes.length} recent scenario runs, ${okEvents}/${totalEvents} events ok (fail rate ${(failRate * 100).toFixed(1)}%)`,
evidence: [
`most recent: ${recent[recent.length - 1]?.run_id ?? "?"}`,
`recent failing sigs: ${recentFailSigs.length > 0 ? recentFailSigs.slice(-3).join(", ") : "none"}`,
],
}];
return findings;
}
async function checkErrorCorrections(_claims: Claim[]): Promise<Finding[]> {
const corrections = await tailJsonl<any>(join(KB_DIR, "error_corrections.jsonl"), TAIL_LINES);
if (corrections.length === 0) return [];
return [{
check: "kb_query",
severity: "info",
summary: `KB: ${corrections.length} error corrections on file (fail→succeed pairs)`,
evidence: [
corrections.length > 0
? `most recent: ${String(corrections[corrections.length - 1]?.sig_hash ?? "?").slice(0, 24)}`
: "none",
],
}];
}
async function checkBotCycles(): Promise<Finding[]> {
let entries: string[] = [];
try { entries = await readdir(BOT_CYCLES_DIR); }
catch { return []; }
const jsonFiles = entries.filter(e => e.endsWith(".json"));
if (jsonFiles.length === 0) return [];
// Sort by mtime desc, take most recent N
const withStat = await Promise.all(
jsonFiles.map(async name => {
try { return { name, mtime: (await stat(join(BOT_CYCLES_DIR, name))).mtimeMs }; }
catch { return { name, mtime: 0 }; }
}),
);
const recent = withStat.sort((a, b) => b.mtime - a.mtime).slice(0, MAX_BOT_CYCLE_FILES);
const outcomes: Record<string, number> = {};
for (const { name } of recent) {
try {
const r = JSON.parse(await readFile(join(BOT_CYCLES_DIR, name), "utf8"));
const o = String(r.outcome ?? "unknown");
outcomes[o] = (outcomes[o] ?? 0) + 1;
} catch { /* skip */ }
}
const summary = Object.entries(outcomes)
.sort((a, b) => b[1] - a[1])
.map(([k, v]) => `${k}=${v}`)
.join(", ");
const failCount = (outcomes["tests_failed"] ?? 0) + (outcomes["apply_failed"] ?? 0) + (outcomes["model_failed"] ?? 0);
return [{
check: "kb_query",
severity: failCount > recent.length / 2 ? "warn" : "info",
summary: `KB: bot recorded ${recent.length} recent cycles — ${summary || "no outcomes parsed"}`,
evidence: [
`dir: ${BOT_CYCLES_DIR}`,
`fail-class total: ${failCount} / ${recent.length}`,
],
}];
}
async function checkObserverStream(): Promise<Finding[]> {
const ops = await tailJsonl<any>(OBSERVER_OPS, TAIL_LINES);
if (ops.length === 0) return [];
const failures = ops.filter(o => o.ok === false).length;
return [{
check: "kb_query",
severity: "info",
summary: `KB: observer stream ${ops.length} recent ops, ${failures} failures`,
evidence: [
`source: ${OBSERVER_OPS}`,
`by source: ${observerBySource(ops)}`,
],
}];
}
function observerBySource(ops: any[]): string {
const c: Record<string, number> = {};
for (const o of ops) {
const s = String(o.source ?? "unknown");
c[s] = (c[s] ?? 0) + 1;
}
return Object.entries(c).sort((a, b) => b[1] - a[1]).map(([k, v]) => `${k}=${v}`).join(", ") || "empty";
}
auditor/checks/static.ts
+159
View File
@@ -0,0 +1,159 @@
// Static diff check — grep-style, no AST, no LLM. Looks for patterns
// that are high-signal evidence of placeholder code.
//
// Findings are severity-graded:
// block — explicit non-impl markers (unimplemented!, todo!,
// panic!("not implemented"), throw new Error("not implemented"))
// warn — TODO / FIXME / XXX / HACK comments on added lines,
// new struct fields with no read-site anywhere in the diff,
// suspiciously-empty function bodies ({ Ok(()) } / {} when
// the commit message claims the fn "implements" something)
// info — hardcoded "test" / "dummy" / "placeholder" strings in
// added lines (could be real, just flag for inspection)
//
// Consumes: raw unified diff text from Gitea.
import type { Finding } from "../types.ts";
// Rust + TypeScript patterns that almost always indicate "this is
// not actually implemented yet."
const BLOCK_PATTERNS: Array<{ re: RegExp; why: string }> = [
{ re: /\bunimplemented!\s*\(/, why: "unimplemented!() macro call" },
{ re: /\btodo!\s*\(/, why: "todo!() macro call" },
{ re: /panic!\s*\(\s*"(?:not implemented|TODO|not yet|unimpl)/i, why: "panic! with not-implemented message" },
{ re: /throw\s+new\s+Error\s*\(\s*['"](?:not implemented|TODO|unimpl)/i, why: "throw Error 'not implemented'" },
];
const WARN_COMMENT_PATTERNS: Array<{ re: RegExp; why: string }> = [
{ re: /^\+.*\/\/\s*(TODO|FIXME|XXX|HACK)\b/i, why: "TODO/FIXME/XXX/HACK comment added" },
{ re: /^\+.*#\s*(TODO|FIXME|XXX|HACK)\b/i, why: "TODO/FIXME/XXX/HACK comment added" },
];
const INFO_HARDCODED_PATTERNS: Array<{ re: RegExp; why: string }> = [
{ re: /"(?:placeholder|dummy|foobar|xxx|replaceme|changeme)"/i, why: "suspicious hardcoded string" },
];
export function runStaticCheck(diff: string): Finding[] {
const findings: Finding[] = [];
// Per-file walk: only look at ADDED lines (prefix '+' but not '+++'
// which is the diff header).
const perFile = splitDiffByFile(diff);
for (const [path, lines] of perFile) {
// Skip diff bookkeeping + pure-delete files
if (!lines.some(l => l.startsWith("+") && !l.startsWith("+++"))) continue;
// The auditor's own check files literally contain the BLOCK
// patterns as regex definitions (BLOCK_PATTERNS in this file,
// prompt examples in inference.ts). Skipping BLOCK scan on these
// specific paths prevents the checker from self-flagging its own
// string literals. WARN/INFO patterns still run — those genuinely
// could indicate problems in the checker's own code (TODO
// comments don't self-define).
const isAuditorCheckerFile = path.startsWith("auditor/checks/") ||
path.startsWith("auditor/fixtures/");
for (let idx = 0; idx < lines.length; idx++) {
const line = lines[idx];
if (!line.startsWith("+") || line.startsWith("+++")) continue;
const added = line.slice(1);
if (!isAuditorCheckerFile) {
for (const { re, why } of BLOCK_PATTERNS) {
if (re.test(added)) {
findings.push({
check: "static",
severity: "block",
summary: `${why} in ${path}`,
evidence: [`${path}:+${idx + 1}: ${added.trim().slice(0, 160)}`],
});
}
}
}
for (const { re, why } of WARN_COMMENT_PATTERNS) {
if (re.test(line)) {
findings.push({
check: "static",
severity: "warn",
summary: `${why} in ${path}`,
evidence: [`${path}:+${idx + 1}: ${added.trim().slice(0, 160)}`],
});
}
}
for (const { re, why } of INFO_HARDCODED_PATTERNS) {
if (re.test(added)) {
findings.push({
check: "static",
severity: "info",
summary: `${why} in ${path}`,
evidence: [`${path}:+${idx + 1}: ${added.trim().slice(0, 160)}`],
});
}
}
}
// "Field added but never read" heuristic — catches exactly the
// Phase 45 DocRef placeholder pattern. Limited to the diff itself:
// we're not doing a full-codebase grep here (too noisy; callers
// elsewhere might exist). The point is: if NEITHER this diff nor
// any other line in the diff reads the field, the PR is shipping
// state without a consumer.
const addedLines = lines.filter(l => l.startsWith("+") && !l.startsWith("+++"))
.map(l => l.slice(1));
const newFields = extractNewFields(addedLines);
for (const field of newFields) {
const readPattern = new RegExp(`[\\.:]\\s*${escape(field)}\\b|\\b${escape(field)}\\s*:`);
// The definition line itself matches readPattern — filter it out
// by requiring at least TWO lines in the diff mention the field
// (one defines, one reads).
const hits = addedLines.filter(l => readPattern.test(l));
if (hits.length < 2) {
findings.push({
check: "static",
severity: "warn",
summary: `field '${field}' added in ${path} but no read-site in the diff — could be placeholder state without a consumer`,
evidence: [`${path}: added '${field}' with no reader; rest of diff has ${hits.length - 1} mentions`],
});
}
}
}
return findings;
}
function splitDiffByFile(diff: string): Map<string, string[]> {
const out = new Map<string, string[]>();
let current: string | null = null;
let buf: string[] = [];
for (const line of diff.split(/\r?\n/)) {
const m = line.match(/^diff --git a\/(\S+) b\/(\S+)/);
if (m) {
if (current) out.set(current, buf);
current = m[2];
buf = [];
continue;
}
buf.push(line);
}
if (current) out.set(current, buf);
return out;
}
// Extract new `pub name: Type,` fields from added lines. Rust syntax.
// Narrowly-scoped: only matches at the start of a trimmed line,
// requires `pub ` prefix, ignores `pub fn` / `pub struct` / etc.
function extractNewFields(addedLines: string[]): string[] {
const fields = new Set<string>();
for (const line of addedLines) {
const t = line.trim();
// pub NAME: Type,
const m = t.match(/^pub\s+(?!fn\b|struct\b|enum\b|mod\b|use\b|trait\b|impl\b|const\b|static\b|type\b)(\w+)\s*:/);
if (m) fields.add(m[1]);
}
return Array.from(fields);
}
function escape(s: string): string {
return s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
}
auditor/claim_parser.ts
+119
View File
@@ -0,0 +1,119 @@
// Claim parser — reads commit messages + PR body, extracts ship-claims.
//
// A "ship-claim" is any phrase that asserts functionality is working,
// tested, complete, or landed. These are the assertions the downstream
// checks (static/dynamic/inference/kb) try to falsify.
//
// Heuristic approach (regex + strength grading) — intentionally NOT
// using an LLM here. Reason: the inference check already asks a cloud
// model "does this match the claim?". The parser's job is to surface
// the claim substrates, not judge them. Over-engineering the parser
// risks false-negatives when the cloud model was going to catch it
// anyway.
import type { Claim, PrSnapshot } from "./types.ts";
// Strong claims: explicit end-to-end + verification vocabulary
const STRONG_PATTERNS: RegExp[] = [
/\bverified\s+(end[- ]to[- ]end|live|in\s+production|against)\b/i,
/\btested\s+(live|end[- ]to[- ]end|against|with)\b/i,
/\bworks\s+(end[- ]to[- ]end|live|in\s+production)\b/i,
/\bproduction[- ]ready\b/i,
/\bfully\s+(functional|wired|working)\b/i,
/\bphase\s+\d+(\.\d+)?\s+(shipped|complete|done|landed)\b/i,
/\bground\s+truth\b/i,
/\bproven\b/i,
];
// Moderate claims: asserted completion or pass but without the strong
// verification qualifier.
const MODERATE_PATTERNS: RegExp[] = [
/\bshipped\b/i,
/\blanded\b/i,
/\bgreen\b/i,
/\b(tests?\s+)?pass(ing|ed)\b/i,
/\bcomplet(e|ed)\b/i,
/\bdone\b/i,
/\bwired\b/i,
/\bfixed\b/i,
/\bworks\b/i,
];
// Weak claims: aspirational or hedged. Usually low-risk but recorded
// for completeness.
const WEAK_PATTERNS: RegExp[] = [
/\bshould\s+work\b/i,
/\bexpected\s+to\b/i,
/\bintended\s+to\b/i,
/\bwill\s+(work|handle|support)\b/i,
/\bprobably\b/i,
];
export interface ParsedClaims {
claims: Claim[];
commits_scanned: number;
}
export function parseClaims(pr: PrSnapshot): ParsedClaims {
const claims: Claim[] = [];
// PR body — every matching line becomes a claim at location "pr_body:N"
if (pr.body) {
scanText(pr.body, "pr_body", pr.head_sha, claims);
}
// Each commit message gets its own scan.
for (const c of pr.commits) {
if (!c.message) continue;
scanText(c.message, `commit:${c.sha.slice(0, 8)}`, c.sha, claims);
}
return { claims, commits_scanned: pr.commits.length };
}
function scanText(text: string, location_prefix: string, commit_sha: string, out: Claim[]): void {
const lines = text.split(/\r?\n/);
for (let i = 0; i < lines.length; i++) {
const line = lines[i];
if (line.length < 3) continue;
// Strong patterns first — if a line matches strong, it's strong,
// don't double-count as moderate.
const strong = firstMatch(line, STRONG_PATTERNS);
if (strong) {
out.push({
text: line.trim().slice(0, 200),
commit_sha,
location: `${location_prefix}:${i + 1}`,
strength: "strong",
});
continue;
}
const moderate = firstMatch(line, MODERATE_PATTERNS);
if (moderate) {
out.push({
text: line.trim().slice(0, 200),
commit_sha,
location: `${location_prefix}:${i + 1}`,
strength: "moderate",
});
continue;
}
const weak = firstMatch(line, WEAK_PATTERNS);
if (weak) {
out.push({
text: line.trim().slice(0, 200),
commit_sha,
location: `${location_prefix}:${i + 1}`,
strength: "weak",
});
}
}
}
function firstMatch(text: string, patterns: RegExp[]): RegExp | null {
for (const p of patterns) {
if (p.test(text)) return p;
}
return null;
}
auditor/fixtures/cli.ts
+49
View File
@@ -0,0 +1,49 @@
// Standalone runner for auditor fixtures. Invoke:
//
// bun run auditor/fixtures/cli.ts hybrid_38_40_45
//
// Prints human-readable per-layer breakdown + JSON result so the
// output can be captured by the dynamic check without re-running.
import { runHybridFixture, type FixtureResult } from "./hybrid_38_40_45.ts";
const fixtureArg = process.argv[2] ?? "hybrid_38_40_45";
let result: FixtureResult;
switch (fixtureArg) {
case "hybrid_38_40_45":
result = await runHybridFixture();
break;
default:
console.error(`unknown fixture: ${fixtureArg}`);
process.exit(2);
}
// Human-readable summary
console.error(""); // blank line to stderr so stdout JSON stays clean
console.error(`─── Fixture: ${result.fixture} ───`);
console.error(` overall: ${result.overall.toUpperCase()}`);
console.error(` shipped: [${result.shipped_phases.join(", ") || "—"}]`);
console.error(` placeholder: [${result.placeholder_phases.join(", ") || "—"}]`);
console.error("");
for (const l of result.layers) {
const mark = l.ok ? "✓" : "✗";
const phaseStr = `Phase ${l.phase}`.padEnd(11);
console.error(` ${mark} ${phaseStr} ${l.layer.padEnd(30)} ${String(l.latency_ms).padStart(6)}ms`);
if (l.ok) {
console.error(` ${l.evidence.slice(0, 200)}`);
} else {
console.error(` ERROR: ${l.error?.slice(0, 240) ?? "unknown"}`);
}
}
console.error("");
for (const n of result.notes) console.error(` note: ${n}`);
console.error("");
// Machine-readable output on stdout.
console.log(JSON.stringify(result, null, 2));
// Exit code reflects overall: 0 pass, 2 partial, 1 fail.
// Dynamic check reads this AND the JSON; partial-pass is treated
// as informative (some layers shipped), not blocking on its own.
process.exit(result.overall === "pass" ? 0 : result.overall === "partial_pass" ? 2 : 1);
auditor/fixtures/hybrid_38_40_45.ts
+335
View File
@@ -0,0 +1,335 @@
// The never-run hybrid test fixture. Exercises the full stack end-to-end
// across Phase 38 (/v1/chat), Phase 40 (Langfuse tracing), Phase 45
// slice 1 (DocRef on playbook), and Phase 45 slice 2 (context7 bridge
// drift detection).
//
// Returns HONEST per-layer results. A layer that's unimplemented or
// broken returns ok=false with a real error; overall verdict flags
// which phases are genuinely shipped vs still placeholder.
//
// Deterministic: always runs the same payload so a run's output is
// comparable across audits. Uses TEST-prefixed names so playbook
// state doesn't get polluted with production-looking fixture data.
//
// Preconditions:
// - gateway up on :3100
// - Python sidecar up on :3200
// - Ollama local + Ollama Cloud key loaded
// - Langfuse up on :3001
// - context7 bridge up on :3900 (manual-start per mcp-server/README)
// If bridge isn't running, that layer reports ok=false with a
// clear error — the fixture doesn't try to auto-start it.
import { readFile } from "node:fs/promises";
const GATEWAY = process.env.LH_GATEWAY_URL ?? "http://localhost:3100";
const LANGFUSE = process.env.LANGFUSE_URL ?? "http://localhost:3001";
const BRIDGE = process.env.CONTEXT7_BRIDGE_URL ?? "http://localhost:3900";
const FIXTURE_ID = "auditor:hybrid_38_40_45:v1";
const TEST_WORKER_NAME = "__auditor_test_worker__";
const STALE_HASH = "stale-hash-for-drift-proof";
// Unique-per-run identifiers so each fixture invocation hits the ADD
// path in upsert_entry (not UPDATE/NOOP on a leftover from a prior
// run). Catches state pollution + avoids interaction with task #12
// (UPDATE branch silently drops doc_refs).
const RUN_NONCE = Date.now().toString(36);
const TEST_WORKER_NAME_VERSIONED = `__auditor_test_worker_${RUN_NONCE}__`;
const TEST_OPERATION_VERSIONED = `TEST: fill: __auditor_${RUN_NONCE}__ x1 in TestCity, TC`;
export interface LayerResult {
layer: string;
phase: string;
ok: boolean;
latency_ms: number;
evidence: string;
error?: string;
// Layer-specific numbers that downstream checks can aggregate.
metrics?: Record<string, number | string | boolean>;
}
export interface FixtureResult {
fixture: string;
ran_at: string;
overall: "pass" | "partial_pass" | "fail";
layers: LayerResult[];
real_numbers: Record<string, number>;
shipped_phases: string[];
placeholder_phases: string[];
notes: string[];
}
// Wraps a layer's async body with timing + structured error capture.
// On throw: returns ok=false with the error string as evidence, so
// downstream verdict code treats it as a layer failure rather than
// a fixture crash.
async function measureLayer(
layer: string,
phase: string,
body: () => Promise<{ evidence: string; metrics?: Record<string, number | string | boolean> }>,
): Promise<LayerResult> {
const t0 = Date.now();
try {
const { evidence, metrics } = await body();
return {
layer, phase, ok: true,
latency_ms: Date.now() - t0,
evidence, metrics,
};
} catch (e) {
return {
layer, phase, ok: false,
latency_ms: Date.now() - t0,
evidence: `FAILED: ${(e as Error).message.slice(0, 200)}`,
error: (e as Error).message,
};
}
}
async function langfuseAuth(): Promise<{ pk: string; sk: string } | null> {
// Default credentials match mcp-server/tracing.ts. Same sources the
// Rust side uses in gateway/src/v1/langfuse_trace.rs.
let pk = process.env.LANGFUSE_PUBLIC_KEY ?? "pk-lf-staffing";
let sk = process.env.LANGFUSE_SECRET_KEY ?? "sk-lf-staffing-secret";
if (!pk || !sk) return null;
return { pk, sk };
}
export async function runHybridFixture(): Promise<FixtureResult> {
const result: FixtureResult = {
fixture: FIXTURE_ID,
ran_at: new Date().toISOString(),
overall: "fail",
layers: [],
real_numbers: {},
shipped_phases: [],
placeholder_phases: [],
notes: [],
};
// ========================================================================
// Layer 1 — Phase 38: POST /v1/chat returns valid OpenAI shape
// ========================================================================
// Captured HERE, immediately before the chat layer runs, so layer 2's
// Langfuse-trace filter uses the actual moment the chat call was
// attempted — not the fixture start time. Earlier draft had a
// meaningless ternary returning result.ran_at on both branches; the
// LLM-Team codereview (2026-04-22) caught this and flagged it as a
// false-negative window on traces created between fixture-start and
// chat-fetch.
const chat_request_sent_ms = Date.now();
const l1 = await measureLayer("phase38_chat", "38", async () => {
const r = await fetch(`${GATEWAY}/v1/chat`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
model: "qwen3.5:latest",
messages: [
{ role: "system", content: "Respond in 8 words or fewer." },
{ role: "user", content: "Name one Docker command." },
],
max_tokens: 60,
temperature: 0.2,
}),
signal: AbortSignal.timeout(60000),
});
if (!r.ok) throw new Error(`gateway ${r.status}: ${await r.text()}`);
const j: any = await r.json();
const content = j.choices?.[0]?.message?.content;
if (typeof content !== "string" || content.length === 0) {
throw new Error(`empty content — think-false default may not be wired: ${JSON.stringify(j).slice(0, 200)}`);
}
return {
evidence: `content="${content.slice(0, 60)}" tokens=${j.usage?.total_tokens}`,
metrics: {
prompt_tokens: j.usage?.prompt_tokens ?? 0,
completion_tokens: j.usage?.completion_tokens ?? 0,
total_tokens: j.usage?.total_tokens ?? 0,
content_nonempty: true,
},
};
});
result.layers.push(l1);
// ========================================================================
// Layer 2 — Phase 40: Langfuse trace lands within 3s
// ========================================================================
// Fire-and-forget tracing means we need a brief sleep before query.
await new Promise(res => setTimeout(res, 2500));
const l2 = await measureLayer("phase40_langfuse_trace", "40", async () => {
const auth = await langfuseAuth();
if (!auth) throw new Error("Langfuse credentials not in env");
const r = await fetch(`${LANGFUSE}/api/public/traces?limit=5&name=v1.chat:ollama`, {
headers: { Authorization: `Basic ${btoa(`${auth.pk}:${auth.sk}`)}` },
signal: AbortSignal.timeout(10000),
});
if (!r.ok) throw new Error(`langfuse ${r.status}: ${await r.text()}`);
const j: any = await r.json();
const items = Array.isArray(j.data) ? j.data : [];
// Filter on the chat-request timestamp captured above. A Langfuse
// trace must be newer than the moment we fired /v1/chat to plausibly
// belong to our request. Using fixture start time (result.ran_at)
// was wrong and could false-negative on slow fixtures.
const recent = items.filter((t: any) => Date.parse(t.timestamp) >= chat_request_sent_ms);
if (recent.length === 0) {
throw new Error(`no v1.chat:ollama trace since ${new Date(chat_request_sent_ms).toISOString()} (${items.length} older traces visible, Langfuse reachable — tracing is not firing)`);
}
const trace = recent[0];
return {
evidence: `trace ${trace.id.slice(0, 8)} name=${trace.name} age=${Date.now() - Date.parse(trace.timestamp)}ms`,
metrics: {
trace_age_ms: Date.now() - Date.parse(trace.timestamp),
trace_latency_reported: Number(trace.latency ?? 0),
recent_trace_count: recent.length,
},
};
});
result.layers.push(l2);
// ========================================================================
// Layer 3 — Phase 45 slice 1: /seed accepts doc_refs and persists them
// ========================================================================
// Seed a playbook with doc_refs referencing Docker at a stale hash.
// Ignore the cleanup concern for v1 — this fixture pollutes the
// in-memory playbook state; operator can retire test entries.
const l3 = await measureLayer("phase45_seed_with_doc_refs", "45.1", async () => {
const r = await fetch(`${GATEWAY}/vectors/playbook_memory/seed`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
operation: TEST_OPERATION_VERSIONED,
approach: "auditor hybrid fixture run",
context: "doc_refs integration test — retire after audit",
endorsed_names: [TEST_WORKER_NAME_VERSIONED],
append: true,
doc_refs: [
{
tool: "docker",
version_seen: "24.0.7",
snippet_hash: STALE_HASH,
source_url: "https://context7.com/docker/docs",
seen_at: new Date().toISOString(),
},
],
}),
signal: AbortSignal.timeout(30000),
});
if (!r.ok) throw new Error(`seed ${r.status}: ${await r.text()}`);
const j: any = await r.json();
// Verify doc_refs actually persisted — read state.json directly.
// That's the canonical source of truth; the seed response may or
// may not echo doc_refs back.
const state_raw = await readFile("/home/profit/lakehouse/data/_playbook_memory/state.json", "utf8");
const state = JSON.parse(state_raw);
const entries = state.entries ?? [];
const ours = entries.find((e: any) => (e.endorsed_names ?? []).includes(TEST_WORKER_NAME_VERSIONED));
if (!ours) throw new Error(`seeded entry not found in state.json (worker ${TEST_WORKER_NAME_VERSIONED} not present)`);
const docRefs = ours.doc_refs ?? [];
if (docRefs.length === 0) {
throw new Error("entry saved but doc_refs field is empty — the field accepted by the API isn't being persisted");
}
return {
evidence: `playbook ${ours.playbook_id.slice(0, 16)} persisted with doc_refs.length=${docRefs.length}; first tool=${docRefs[0]?.tool} hash=${docRefs[0]?.snippet_hash}`,
metrics: {
doc_refs_stored: docRefs.length,
entries_after: j.entries_after ?? -1,
playbook_id: ours.playbook_id,
mode: j.outcome?.mode ?? "unknown",
},
};
});
result.layers.push(l3);
// ========================================================================
// Layer 4 — Phase 45 slice 2: context7 bridge detects drift vs stale hash
// ========================================================================
const l4 = await measureLayer("phase45_bridge_diff", "45.2", async () => {
// Health check first — if bridge isn't running, fail with a clear
// message rather than mysterious connection refused.
try {
const h = await fetch(`${BRIDGE}/health`, { signal: AbortSignal.timeout(3000) });
if (!h.ok) throw new Error(`bridge /health ${h.status}`);
} catch (e) {
throw new Error(`context7 bridge at ${BRIDGE} is not reachable (bridge is manual-start; run 'bun run mcp-server/context7_bridge.ts'): ${(e as Error).message}`);
}
const r = await fetch(`${BRIDGE}/docs/docker/diff?since=${encodeURIComponent(STALE_HASH)}`, {
signal: AbortSignal.timeout(30000),
});
if (!r.ok) throw new Error(`bridge diff ${r.status}: ${await r.text()}`);
const j: any = await r.json();
if (j.drifted !== true) {
throw new Error(`expected drifted=true against ${STALE_HASH}; got ${JSON.stringify(j)}`);
}
return {
evidence: `bridge confirms drift: current=${j.current_snippet_hash} previous=${j.previous_snippet_hash} upstream_updated=${j.last_updated_upstream}`,
metrics: {
drifted: true,
current_hash_length: String(j.current_snippet_hash ?? "").length,
library_id: j.library_id,
},
};
});
result.layers.push(l4);
// ========================================================================
// Layer 5 — Phase 45 slice 3: /doc_drift/check/{id} endpoint
// (EXPECTED TO FAIL — endpoint unimplemented)
// ========================================================================
const l5 = await measureLayer("phase45_slice3_drift_flag", "45.3", async () => {
const pid = String(l3.metrics?.playbook_id ?? "");
if (!pid) throw new Error("layer 3 did not produce a playbook_id to check");
const r = await fetch(`${GATEWAY}/vectors/playbook_memory/doc_drift/check/${encodeURIComponent(pid)}`, {
method: "POST",
signal: AbortSignal.timeout(10000),
});
if (r.status === 404 || r.status === 405) {
// This is the HONEST signal: the endpoint doesn't exist yet.
// Fail this layer with a clear marker — not a silent pass.
throw new Error(`endpoint unimplemented (${r.status}) — Phase 45 slice 3 is still placeholder. doc_refs are persisted (layer 3) and the bridge can answer drift questions (layer 4), but nothing ties them together into a playbook flag yet.`);
}
if (!r.ok) throw new Error(`drift/check ${r.status}: ${await r.text()}`);
const j: any = await r.json();
// If the endpoint DID exist, assert the flag got set.
if (!j.flagged) {
throw new Error(`endpoint responded but flag not set: ${JSON.stringify(j).slice(0, 200)}`);
}
return {
evidence: `playbook flagged: ${JSON.stringify(j).slice(0, 160)}`,
metrics: { flagged: true },
};
});
result.layers.push(l5);
// ========================================================================
// Verdict assembly
// ========================================================================
for (const l of result.layers) {
if (l.ok) result.shipped_phases.push(l.phase);
else result.placeholder_phases.push(l.phase);
result.real_numbers[`${l.layer}_latency_ms`] = l.latency_ms;
if (l.metrics) {
for (const [k, v] of Object.entries(l.metrics)) {
if (typeof v === "number") result.real_numbers[`${l.layer}.${k}`] = v;
}
}
}
const anyFail = result.placeholder_phases.length > 0;
const allPass = result.layers.every(l => l.ok);
result.overall = allPass ? "pass" : (result.shipped_phases.length > 0 ? "partial_pass" : "fail");
if (!l5.ok) {
result.notes.push(
"Phase 45 slice 3 (doc_drift/check endpoint) is the expected-fail layer. " +
"Layer 5 failing with a 404/405 is the CORRECT honest signal — don't mask this " +
"to get a green result. When slice 3 ships, layer 5 flips to ok=true automatically."
);
}
if (anyFail) {
result.notes.push(
`${result.placeholder_phases.length} layer(s) failed. Check evidence per layer for specifics.`
);
}
return result;
}
auditor/gitea.ts
+145
View File
@@ -0,0 +1,145 @@
// Gitea API client. Minimal surface — only what the auditor needs:
// list open PRs, get commits + files for a PR, fetch a diff, post a
// commit status, post a review.
//
// Auth: reads PAT from ~/.git-credentials (set up by the credential
// helper flow in 2026-04-22 session). Gitea's "token" auth scheme
// matches what `git fetch` is already using.
import { readFile } from "node:fs/promises";
import type { PrSnapshot } from "./types.ts";
const HOST = process.env.GITEA_HOST ?? "https://git.agentview.dev";
const OWNER = "profit";
const REPO = "lakehouse";
const CRED_FILE = "/home/profit/.git-credentials";
let cachedPat: string | null = null;
async function getPat(): Promise<string> {
if (cachedPat) return cachedPat;
const raw = await readFile(CRED_FILE, "utf8");
for (const line of raw.split("\n")) {
const m = line.match(/^https:\/\/[^:]+:([^@]+)@git\.agentview\.dev/);
if (m) { cachedPat = m[1]; return m[1]; }
}
throw new Error(`no Gitea PAT in ${CRED_FILE}`);
}
async function giteaFetch(path: string, init: RequestInit = {}): Promise<Response> {
const pat = await getPat();
const url = `${HOST}/api/v1${path}`;
const headers = new Headers(init.headers);
headers.set("Authorization", `token ${pat}`);
if (init.body && !headers.has("content-type")) {
headers.set("content-type", "application/json");
}
return fetch(url, { ...init, headers, signal: AbortSignal.timeout(20000) });
}
export async function listOpenPrs(): Promise<PrSnapshot[]> {
const r = await giteaFetch(`/repos/${OWNER}/${REPO}/pulls?state=open&page=1&limit=50`);
if (!r.ok) throw new Error(`listOpenPrs ${r.status}: ${await r.text()}`);
const rows = (await r.json()) as any[];
return Promise.all(rows.map(row => snapshotFromPr(row)));
}
export async function getPrSnapshot(num: number): Promise<PrSnapshot> {
const r = await giteaFetch(`/repos/${OWNER}/${REPO}/pulls/${num}`);
if (!r.ok) throw new Error(`getPr ${num} ${r.status}: ${await r.text()}`);
return snapshotFromPr((await r.json()) as any);
}
async function snapshotFromPr(row: any): Promise<PrSnapshot> {
const num = row.number;
const commitsResp = await giteaFetch(`/repos/${OWNER}/${REPO}/pulls/${num}/commits`);
const commits = commitsResp.ok ? ((await commitsResp.json()) as any[]) : [];
const filesResp = await giteaFetch(`/repos/${OWNER}/${REPO}/pulls/${num}/files`);
const files = filesResp.ok ? ((await filesResp.json()) as any[]) : [];
return {
number: num,
head_sha: row.head?.sha ?? "",
base_sha: row.base?.sha ?? "",
title: row.title ?? "",
body: row.body ?? "",
state: row.state === "open" ? "open" : (row.merged ? "merged" : "closed"),
author: row.user?.login ?? "",
commits: commits.map(c => ({
sha: (c.sha ?? "").slice(0, 12),
message: c.commit?.message ?? "",
author: c.commit?.author?.name ?? "",
})),
files: files.map(f => ({
path: f.filename ?? "",
additions: f.additions ?? 0,
deletions: f.deletions ?? 0,
})),
};
}
/// Returns the unified diff text of the PR. Used by static checks.
export async function getPrDiff(num: number): Promise<string> {
const r = await giteaFetch(`/repos/${OWNER}/${REPO}/pulls/${num}.diff`);
if (!r.ok) throw new Error(`getDiff ${num} ${r.status}: ${await r.text()}`);
return await r.text();
}
/// Hard-block mechanism: post a failing commit status on the PR head
/// SHA. Branch protection (if enabled on `main`) treats this as a
/// required-check fail and prevents merge. The description is shown
/// in the Gitea UI next to the red X.
export async function postCommitStatus(args: {
sha: string;
state: "success" | "pending" | "failure" | "error";
context: string;
description: string;
target_url?: string;
}): Promise<void> {
const r = await giteaFetch(`/repos/${OWNER}/${REPO}/statuses/${args.sha}`, {
method: "POST",
body: JSON.stringify({
state: args.state,
context: args.context,
description: args.description.slice(0, 140),
target_url: args.target_url ?? "",
}),
});
if (!r.ok) throw new Error(`postCommitStatus ${r.status}: ${await r.text()}`);
}
/// Post a review comment. Gitea typically blocks self-review
/// (author posting a review on their own PR). Prefer
/// `postIssueComment` when running with the author's PAT.
export async function postReview(args: {
pr_number: number;
commit_id: string;
body: string;
event: "APPROVE" | "REQUEST_CHANGES" | "COMMENT";
}): Promise<void> {
const r = await giteaFetch(`/repos/${OWNER}/${REPO}/pulls/${args.pr_number}/reviews`, {
method: "POST",
body: JSON.stringify({
commit_id: args.commit_id,
body: args.body,
event: args.event,
}),
});
if (!r.ok) throw new Error(`postReview ${r.status}: ${await r.text()}`);
}
/// Plain issue comment. Works for the auditor's own PAT because
/// Gitea allows authors to comment on their own PRs (just not
/// review them). Auditor uses this for the reasoning body; the
/// actual block signal is the commit status.
export async function postIssueComment(args: {
pr_number: number;
body: string;
}): Promise<{ id: number; html_url: string }> {
const r = await giteaFetch(`/repos/${OWNER}/${REPO}/issues/${args.pr_number}/comments`, {
method: "POST",
body: JSON.stringify({ body: args.body }),
});
if (!r.ok) throw new Error(`postIssueComment ${r.status}: ${await r.text()}`);
const j = await r.json() as any;
return { id: j.id, html_url: j.html_url };
}
auditor/index.ts
+147
View File
@@ -0,0 +1,147 @@
// Auditor poller — the top-level entry. Polls Gitea for open PRs on
// a fixed interval, dedupes by head SHA, runs audit + posts verdict
// for each new (pr, sha) pair.
//
// Run manually:
// bun run auditor/index.ts
//
// Stop:
// touch auditor.paused (skips next cycle)
// pkill -f auditor/index.ts (kills in-flight)
//
// State:
// data/_auditor/state.json — last-audited SHA per PR
// data/_auditor/verdicts/{id}.json — per-run verdict records
//
// This entry runs forever. A systemd unit would wrap it once the
// workflow is trusted (same pattern as mcp-server, observer).
import { readFile, writeFile, mkdir, access } from "node:fs/promises";
import { listOpenPrs } from "./gitea.ts";
import { auditPr } from "./audit.ts";
const POLL_INTERVAL_MS = 90_000; // 90s — enough budget for audit runs to complete
const PAUSE_FILE = "/home/profit/lakehouse/auditor.paused";
const STATE_FILE = "/home/profit/lakehouse/data/_auditor/state.json";
interface State {
// Map: PR number → last-audited head SHA. Lets us dedupe audits
// across restarts (poller can crash/restart without re-auditing
// all open PRs from scratch).
last_audited: Record<string, string>;
started_at: string;
cycles_total: number;
cycles_skipped_paused: number;
audits_run: number;
last_cycle_at?: string;
}
async function fileExists(path: string): Promise<boolean> {
try { await access(path); return true; } catch { return false; }
}
async function loadState(): Promise<State> {
try {
const raw = await readFile(STATE_FILE, "utf8");
const s = JSON.parse(raw);
return {
last_audited: s.last_audited ?? {},
started_at: s.started_at ?? new Date().toISOString(),
cycles_total: s.cycles_total ?? 0,
cycles_skipped_paused: s.cycles_skipped_paused ?? 0,
audits_run: s.audits_run ?? 0,
last_cycle_at: s.last_cycle_at,
};
} catch {
return {
last_audited: {},
started_at: new Date().toISOString(),
cycles_total: 0,
cycles_skipped_paused: 0,
audits_run: 0,
};
}
}
async function saveState(s: State): Promise<void> {
await mkdir("/home/profit/lakehouse/data/_auditor", { recursive: true });
await writeFile(STATE_FILE, JSON.stringify(s, null, 2));
}
async function runCycle(state: State): Promise<State> {
state.cycles_total += 1;
state.last_cycle_at = new Date().toISOString();
if (await fileExists(PAUSE_FILE)) {
state.cycles_skipped_paused += 1;
console.log(`[auditor] cycle ${state.cycles_total}: paused (touch ${PAUSE_FILE} exists)`);
return state;
}
let prs;
try {
prs = await listOpenPrs();
} catch (e) {
console.error(`[auditor] listOpenPrs failed: ${(e as Error).message}`);
return state;
}
console.log(`[auditor] cycle ${state.cycles_total}: ${prs.length} open PR(s)`);
for (const pr of prs) {
const last = state.last_audited[String(pr.number)];
if (last === pr.head_sha) {
console.log(`[auditor] skip PR #${pr.number} (SHA ${pr.head_sha.slice(0, 8)} already audited)`);
continue;
}
console.log(`[auditor] audit PR #${pr.number} (${pr.head_sha.slice(0, 8)}) — ${pr.title.slice(0, 60)}`);
try {
// Skip dynamic by default: it mutates live playbook state and
// re-runs on every PR update would pollute quickly. Operator
// can run dynamic via `bun run auditor/fixtures/cli.ts` manually
// OR set LH_AUDITOR_RUN_DYNAMIC=1 to opt in.
const run_dynamic = process.env.LH_AUDITOR_RUN_DYNAMIC === "1";
const verdict = await auditPr(pr, {
skip_dynamic: !run_dynamic,
skip_inference: process.env.LH_AUDITOR_SKIP_INFERENCE === "1",
});
console.log(`[auditor] verdict=${verdict.overall} findings=${verdict.metrics.findings_total} (block=${verdict.metrics.findings_block} warn=${verdict.metrics.findings_warn})`);
state.last_audited[String(pr.number)] = pr.head_sha;
state.audits_run += 1;
} catch (e) {
console.error(`[auditor] audit failed: ${(e as Error).message}`);
}
}
return state;
}
async function main(): Promise<void> {
console.log(`[auditor] starting poller — interval ${POLL_INTERVAL_MS / 1000}s`);
console.log(`[auditor] pause file: ${PAUSE_FILE}`);
console.log(`[auditor] state file: ${STATE_FILE}`);
let state = await loadState();
console.log(`[auditor] loaded state: ${Object.keys(state.last_audited).length} PRs previously audited, ${state.cycles_total} cycles so far`);
// Single-shot mode for CLI testing: `bun run auditor/index.ts --once`
const once = process.argv.includes("--once");
if (once) {
state = await runCycle(state);
await saveState(state);
console.log(`[auditor] single-shot complete. total audits: ${state.audits_run}`);
return;
}
// Loop.
while (true) {
state = await runCycle(state);
await saveState(state);
await new Promise(res => setTimeout(res, POLL_INTERVAL_MS));
}
}
main().catch(e => {
console.error("[auditor] fatal:", e);
process.exit(1);
});
auditor/policy.ts
+62
View File
@@ -0,0 +1,62 @@
// ═══════════════════════════════════════════════════════════════════
// YOU WRITE THIS FILE. Policy decides what blocks vs what's a comment.
// Defaults are opinionated on the "stop clicking past placeholder"
// side — easier to loosen than to tighten when you're watching the
// auditor behave in live PRs.
// ═══════════════════════════════════════════════════════════════════
import type { Finding, Verdict } from "./types.ts";
/// Translate the four-check output into a single verdict. This is the
/// single pane of glass the auditor operates on — tune thresholds here.
export function assembleVerdict(
findings: Finding[],
metrics: Record<string, number>,
pr_number: number,
head_sha: string,
): Verdict {
const blocking = findings.filter(f => f.severity === "block");
const warning = findings.filter(f => f.severity === "warn");
let overall: Verdict["overall"];
let one_liner: string;
if (blocking.length > 0) {
overall = "block";
one_liner = `${blocking.length} blocking issue${blocking.length > 1 ? "s" : ""}: ${blocking[0].summary}`;
} else if (warning.length >= 3) {
// Three or more warnings is a block — death by a thousand cuts.
overall = "request_changes";
one_liner = `${warning.length} warnings — see review`;
} else if (warning.length > 0) {
overall = "request_changes";
one_liner = warning[0].summary;
} else {
overall = "approve";
one_liner = `all checks passed (${findings.length} findings, all info)`;
}
return {
pr_number,
head_sha,
audited_at: new Date().toISOString(),
overall,
findings,
metrics,
one_liner,
};
}
/// Which strength-of-claim warrants which severity when evidence is
/// weak? A "Phase X shipped" claim with zero integration tests is a
/// blocker. A "should work" claim with no test is a warn.
export function severityFromClaimEvidence(
claim_strength: "weak" | "moderate" | "strong",
evidence_grade: "none" | "partial" | "full",
): "info" | "warn" | "block" {
if (evidence_grade === "full") return "info";
if (claim_strength === "strong" && evidence_grade === "none") return "block";
if (claim_strength === "strong" && evidence_grade === "partial") return "warn";
if (claim_strength === "moderate" && evidence_grade === "none") return "warn";
return "info";
}
auditor/types.ts
+65
View File
@@ -0,0 +1,65 @@
// Shared types for the claim-auditor. Every field exists for a reason;
// if something can't be verified from a check, it goes into `evidence`
// so the verdict is inspectable, not a black box.
export type CheckKind = "static" | "dynamic" | "inference" | "kb_query";
export type Severity = "info" | "warn" | "block";
export interface Claim {
// Verbatim phrase that raised the claim — e.g. "Phase 38 shipped",
// "verified end-to-end", "works after restart". Used as the "what
// does the author assert" input to downstream checks.
text: string;
// Where it came from. `commit_sha` is the short hash; `location`
// is a file:line for in-diff claims, or "pr_body" / "commit_message".
commit_sha: string;
location: string;
// Heuristic rating of how strong the claim is. "green+tested"
// is strong; "should work" is weak. Drives sensitivity — stronger
// claims get harder-blocked on weak evidence.
strength: "weak" | "moderate" | "strong";
}
export interface Finding {
check: CheckKind;
severity: Severity;
claim_text?: string;
// Free-form short description: "field added but never read", "no
// test covers this code path", "cloud model says placeholder".
summary: string;
// Concrete evidence: file paths, line numbers, log excerpts, test
// output, cloud-model verdict. No handwaving.
evidence: string[];
}
export interface Verdict {
pr_number: number;
head_sha: string;
audited_at: string;
overall: "approve" | "request_changes" | "block";
findings: Finding[];
// Real numbers that downstream policy can gate on. e.g. if the
// hybrid test produced latency numbers or token counts, they
// surface here so /auditor/history is queryable.
metrics: Record<string, number>;
// Short one-line justification for the `overall` verdict. What
// gets posted as the commit-status description in Gitea (max 140
// chars) must fit here.
one_liner: string;
}
export interface PrSnapshot {
number: number;
head_sha: string;
base_sha: string;
title: string;
body: string;
state: "open" | "closed" | "merged";
author: string;
// Array of commit messages in the PR (not diffs — those are
// fetched on-demand per-check).
commits: Array<{ sha: string; message: string; author: string }>;
// File paths touched by the PR, with lines-added / lines-removed.
files: Array<{ path: string; additions: number; deletions: number }>;
}
bot/README.md
+127
View File
@@ -0,0 +1,127 @@
# Lakehouse PR-Bot
A local sub-agent that reads the PRD, asks a cloud model for a small change
proposal, applies it, runs tests, and opens a draft PR on Gitea. Manual-only
right now — no systemd, no cron. Run one cycle at a time and watch.
## Run one cycle
```bash
cd /home/profit/lakehouse
bun run bot/cycle.ts
```
Prerequisites:
- Working tree must be clean (`git status` shows nothing). The bot refuses on a dirty tree so its changes don't mix with in-flight work.
- Sidecar running on :3200, gateway on :3100, observer on :3800.
- At least one PRD line tagged `[bot-eligible]` (see below).
- Gitea PAT configured in `~/.git-credentials` (already set up).
## Tag a gap as bot-eligible
Add `[bot-eligible]` to any PRD line the bot is allowed to work on:
```md
- [ ] Add a unit test for parse_city_state covering "South Bend, IN" edge case [bot-eligible]
```
The bot scans `docs/PRD.md` for these tags. Each tagged line becomes a candidate.
Start small — one tag at a time — until you trust the loop.
## Stop it mid-cycle
Create the pause file:
```bash
touch /home/profit/lakehouse/bot.paused
```
The next `bot/cycle.ts` invocation exits immediately with `skipped_pause`.
(It does NOT kill a cycle already in-flight — use `Ctrl-C` or `pkill -f bot/cycle.ts` for that.)
## Budget
- **20 cloud calls/day, 160k tokens/day** (hard ceiling, see `bot/cost.ts`).
- Tracked in `data/_bot/cost-YYYY-MM-DD.json`.
- Resets at UTC midnight.
## Cycle outcomes
Every run writes a result to `data/_bot/cycles/{cycle_id}.json` and POSTs an event to the observer on :3800.
Possible outcomes:
| Outcome | Meaning |
|---|---|
| `ok` | PR opened |
| `cycle_noop` | proposal applied, every file was **identical to current content** (Mem0 NOOP); no test run, no PR |
| `skipped_pause` | pause file present |
| `skipped_cost` | daily budget exhausted |
| `skipped_policy` | `policy.shouldRunCycle` said no |
| `skipped_dirty_tree` | uncommitted changes present |
| `skipped_no_gap` | no `[bot-eligible]` tags in PRD |
| `model_failed` | sidecar/cloud model errored or returned unparseable JSON |
| `proposal_rejected` | `policy.scoreProposal` rejected it (size, path, etc.) |
| `apply_failed` | file write errored |
| `tests_failed` | cargo or bun test red |
| `pr_skipped_by_policy` | green tests, but `policy.shouldOpenPR` said no |
| `pr_failed` | Gitea API or git push errored |
## Mem0-aligned apply semantics
`apply.ts` categorizes every proposed file into one of three modes:
| Mode | Trigger | Action |
|---|---|---|
| **ADD** | `is_new: true` and file doesn't exist | Create the file |
| **UPDATE** | `is_new: false`, file exists, content **differs** from current | Overwrite |
| **NOOP** | `is_new: false`, file exists, content **matches** current exactly | Skip (no write, no diff) |
If every file in the proposal is NOOP, the cycle short-circuits to `cycle_noop` **before** running tests or opening a PR. Mismatched shapes (`is_new:true` but file exists, or `is_new:false` but file missing) become `apply_failed` — model state confusion is surfaced, not papered over.
PR bodies report the three counts separately so reviewers see what actually changed vs. what was confirmed identical.
## How the bot compounds over cycles
Every finished cycle persists a `CycleResult` to `data/_bot/cycles/{cycle_id}.json`. At the **start** of the next cycle, `bot/kb.ts::loadHistory(gap_id)` scans that directory, filters to prior cycles on the same gap, and returns the five most recent outcomes (`ok`, `tests_failed`, `proposal_rejected`, `cycle_noop`, etc.) with their reasons and touched files.
Those outcomes are summarized into a compact block and **injected into the cloud prompt** before asking for a new proposal. The model sees things like:
```
Prior attempts on this gap (3 most recent):
- 2026-04-22 03:15 UTC — tests_failed
reason: cargo test -p vectord::lance failed on field_type_coerce
- 2026-04-22 02:48 UTC — proposal_rejected
reason: touched forbidden path (docs/ADR-): docs/ADR-019-update.md
- 2026-04-22 01:23 UTC — ok PR: https://git.agentview.dev/profit/lakehouse/pulls/142
reason: PR #142 opened
Learn from these: build on what worked, avoid paths that failed.
```
This is the same compounding pattern `scenario.ts` uses via `kb.loadRecommendation` — the bot's cycles are the bot's memory. No embedding, no separate jsonl, no cross-run orchestration required. First cycle on a new gap skips the block cleanly.
The observer events POSTed on every cycle carry the same data, so `GET :3800/stats?source=bot` aggregates "% of bot cycles on similar gaps that landed a PR" without extra plumbing.
## Where YOU edit
`bot/policy.ts` — four small functions that define what the bot does. The rest
(`propose.ts`, `apply.ts`, `test.ts`, `pr.ts`, `cycle.ts`, `kb.ts`) is mechanical
orchestration — you shouldn't need to touch it unless a pipeline changes.
One policy upgrade the history opens up: in `shouldRunCycle`, you can now bail
out if the same gap has failed N times in a row. Example addition:
```ts
import { loadHistory, statsFor } from "./kb.ts";
// inside shouldRunCycle — but you'd need the gap, which is picked later.
// A more natural place is scoreProposal: if prior N failures on this
// gap's path+summary pattern, reject before testing.
```
## Hard-coded guardrails (not in policy — can't be disabled)
- Bot **never deletes files** (`apply.ts` has no delete path).
- Bot **never touches** `.git/`, `secrets`, `lakehouse.toml`, `docs/ADR-*`, `docs/DECISIONS.md`, `docs/PRD.md`, `/etc/`, `/root/`, `Cargo.lock`.
- All paths validated for traversal (`path/../`) and repo-escape.
- PRs always open as **draft** — never auto-merge.
- Budget check is before the policy function runs — no way to override the daily cap from `policy.ts`.
bot/apply.ts
+86
View File
@@ -0,0 +1,86 @@
// Apply a proposal to disk with Mem0-aligned ADD / UPDATE / NOOP
// semantics. Whole-file writes only — never deletes. All paths
// validated repo-relative + path-traversal-free.
//
// Three outcomes per file:
// ADD — file didn't exist, new content written (is_new must be true)
// UPDATE — file existed, content differs from current, overwritten
// NOOP — file existed, content identical to current, nothing written
//
// The NOOP case is important: a cloud proposal that "re-derives" an
// existing file shouldn't churn the repo or burn test cycles.
import { writeFile, readFile, mkdir, access } from "node:fs/promises";
import { dirname, resolve, relative } from "node:path";
import type { ApplyOutcome, Proposal } from "./types.ts";
const REPO_ROOT = "/home/profit/lakehouse";
export async function applyProposal(p: Proposal): Promise<ApplyOutcome> {
const added: string[] = [];
const updated: string[] = [];
const noop: string[] = [];
const errors: string[] = [];
for (const f of p.files) {
const err = validatePath(f.path);
if (err) {
errors.push(`${f.path}: ${err}`);
continue;
}
const abs = resolve(REPO_ROOT, f.path);
try {
const exists = await fileExists(abs);
if (f.is_new && exists) {
errors.push(`${f.path}: is_new=true but file exists`);
continue;
}
if (!f.is_new && !exists) {
// Proposal claims to update a file that doesn't exist.
// Refuse rather than silently ADD — model is confused about
// repo state, and we want that surfaced, not papered over.
errors.push(`${f.path}: is_new=false but file does not exist`);
continue;
}
if (!f.is_new) {
// UPDATE candidate — compare to detect NOOP.
const current = await readFile(abs, "utf8");
if (current === f.content) {
noop.push(f.path);
continue;
}
await writeFile(abs, f.content);
updated.push(f.path);
} else {
// ADD — file doesn't exist yet.
await mkdir(dirname(abs), { recursive: true });
await writeFile(abs, f.content);
added.push(f.path);
}
} catch (e) {
errors.push(`${f.path}: ${(e as Error).message}`);
}
}
return { added, updated, noop, errors };
}
function validatePath(p: string): string | null {
if (!p) return "empty path";
if (p.startsWith("/")) return "must be repo-relative";
if (p.startsWith("..") || p.includes("/../")) return "path traversal";
const abs = resolve(REPO_ROOT, p);
const rel = relative(REPO_ROOT, abs);
if (rel.startsWith("..")) return "resolves outside repo";
return null;
}
async function fileExists(abs: string): Promise<boolean> {
try {
await access(abs);
return true;
} catch {
return false;
}
}
bot/cost.ts
+52
View File
@@ -0,0 +1,52 @@
// Daily cost tracker. Resets at UTC midnight by keying the file by date.
// Hard ceilings are the policy surface — the bot refuses to call the
// cloud once the budget is exhausted for the day.
import { readFile, writeFile, mkdir } from "node:fs/promises";
import { join } from "node:path";
import type { CostState } from "./types.ts";
const COST_DIR = "/home/profit/lakehouse/data/_bot";
export const DAILY_CALLS_BUDGET = 20;
export const DAILY_TOKENS_BUDGET = 8000 * DAILY_CALLS_BUDGET; // 160k tokens/day ceiling
function todayUtc(): string {
return new Date().toISOString().slice(0, 10);
}
function costFile(date: string): string {
return join(COST_DIR, `cost-${date}.json`);
}
export async function readCost(): Promise<CostState> {
const date = todayUtc();
try {
const raw = await readFile(costFile(date), "utf8");
return JSON.parse(raw) as CostState;
} catch {
return { date, calls: 0, tokens: 0 };
}
}
export async function recordCost(calls: number, tokens: number): Promise<CostState> {
await mkdir(COST_DIR, { recursive: true });
const current = await readCost();
const updated: CostState = {
date: current.date,
calls: current.calls + calls,
tokens: current.tokens + tokens,
};
await writeFile(costFile(updated.date), JSON.stringify(updated, null, 2));
return updated;
}
export function budgetCheck(c: CostState): { ok: boolean; reason: string } {
if (c.calls >= DAILY_CALLS_BUDGET) {
return { ok: false, reason: `daily call budget exhausted (${c.calls}/${DAILY_CALLS_BUDGET})` };
}
if (c.tokens >= DAILY_TOKENS_BUDGET) {
return { ok: false, reason: `daily token budget exhausted (${c.tokens}/${DAILY_TOKENS_BUDGET})` };
}
return { ok: true, reason: `${c.calls}/${DAILY_CALLS_BUDGET} calls, ${c.tokens}/${DAILY_TOKENS_BUDGET} tokens used today` };
}
bot/cycle.ts
+345
View File
@@ -0,0 +1,345 @@
// Single-cycle orchestrator. This is the manual entry point:
// bun run bot/cycle.ts
//
// Steps (each step can short-circuit with a clear outcome):
// 1. Pause-file check → skipped_pause
// 2. Cost budget check → skipped_cost
// 3. Gather CycleContext (dirty tree, last cycle, autotune status)
// 4. policy.shouldRunCycle → skipped_policy
// 5. findGaps + policy.pickGap → skipped_no_gap
// 6. generateProposal → model_failed
// 7. policy.scoreProposal → proposal_rejected
// 8. git checkout new branch
// 9. applyProposal → apply_failed
// 10. runTests → tests_failed
// 11. policy.shouldOpenPR → pr_skipped_by_policy
// 12. git commit + push
// 13. openPr → pr_failed / ok
// 14. recordCost + postEvent (every outcome)
// 15. write cycle result to data/_bot/cycles/{cycle_id}.json
import { readFile, writeFile, mkdir, access } from "node:fs/promises";
import { createHash, randomUUID } from "node:crypto";
import { spawnSync } from "node:child_process";
import { join } from "node:path";
import type { CycleContext, CycleResult, CycleOutcome, Proposal } from "./types.ts";
import * as policy from "./policy.ts";
import { readCost, recordCost, budgetCheck, DAILY_CALLS_BUDGET, DAILY_TOKENS_BUDGET } from "./cost.ts";
import { findGaps, generateProposal } from "./propose.ts";
import { applyProposal } from "./apply.ts";
import { runTests } from "./test.ts";
import { openPr } from "./pr.ts";
import { postEvent } from "./observer.ts";
import { loadHistory, summarizeHistory, statsFor } from "./kb.ts";
const REPO_ROOT = "/home/profit/lakehouse";
const PAUSE_FILE = `${REPO_ROOT}/bot.paused`;
const CYCLES_DIR = `${REPO_ROOT}/data/_bot/cycles`;
const GATEWAY_URL = process.env.LH_GATEWAY_URL ?? "http://localhost:3100";
function log(msg: string) { console.log(`[bot] ${msg}`); }
async function fileExists(p: string): Promise<boolean> {
try { await access(p); return true; } catch { return false; }
}
function git(args: string[]): { code: number; stdout: string; stderr: string } {
const r = spawnSync("git", args, { cwd: REPO_ROOT, encoding: "utf8" });
return { code: r.status ?? -1, stdout: r.stdout ?? "", stderr: r.stderr ?? "" };
}
async function gatherContext(cost: { calls: number; tokens: number }): Promise<CycleContext> {
// Last cycle: newest file in CYCLES_DIR.
let lastCycleAt: string | null = null;
let lastCycleGapId: string | null = null;
try {
const { readdir, stat } = await import("node:fs/promises");
const entries = await readdir(CYCLES_DIR).catch(() => [] as string[]);
let newestTs = 0;
let newestPath: string | null = null;
for (const e of entries) {
if (!e.endsWith(".json")) continue;
const p = join(CYCLES_DIR, e);
const s = await stat(p);
if (s.mtimeMs > newestTs) { newestTs = s.mtimeMs; newestPath = p; }
}
if (newestPath) {
const r = JSON.parse(await readFile(newestPath, "utf8")) as CycleResult;
lastCycleAt = r.started_at;
lastCycleGapId = r.gap?.id ?? null;
}
} catch {}
// Autotune status — if the agent is busy, skip this cycle to avoid
// contention on shared Ollama GPU memory.
let autotuneBusy = false;
try {
const r = await fetch(`${GATEWAY_URL}/vectors/agent/status`, { signal: AbortSignal.timeout(2000) });
if (r.ok) {
const j = await r.json() as any;
autotuneBusy = Boolean(j.running) && (j.queue_depth ?? 0) > 0;
}
} catch {}
// Dirty tree check.
const status = git(["status", "--porcelain"]);
const workingTreeDirty = status.code === 0 && status.stdout.trim().length > 0;
return {
startedAt: new Date().toISOString(),
dailyCallsUsed: cost.calls,
dailyCallsBudget: DAILY_CALLS_BUDGET,
dailyTokensUsed: cost.tokens,
dailyTokensBudget: DAILY_TOKENS_BUDGET,
lastCycleAt,
lastCycleGapId,
autotuneBusy,
workingTreeDirty,
};
}
async function persistResult(res: CycleResult): Promise<void> {
await mkdir(CYCLES_DIR, { recursive: true });
await writeFile(join(CYCLES_DIR, `${res.cycle_id}.json`), JSON.stringify(res, null, 2));
}
function sigHash(gap_id: string | null, proposalSummary: string): string {
const h = createHash("sha256");
h.update(gap_id ?? "no-gap");
h.update("|");
h.update(proposalSummary);
return h.digest("hex").slice(0, 16);
}
function emit(res: CycleResult): Promise<void> {
return postEvent({
source: "bot",
cycle_id: res.cycle_id,
sig_hash: sigHash(res.gap?.id ?? null, res.proposal?.summary ?? ""),
event_kind: res.outcome,
ok: res.outcome === "ok",
staffer_id: res.proposal?.model_used,
turns: res.cloud_calls,
duration_ms: new Date(res.ended_at).getTime() - new Date(res.started_at).getTime(),
extra: {
reason: res.reason,
pr_url: res.prUrl,
files_added: res.filesAdded,
files_updated: res.filesUpdated,
files_noop: res.filesNoop,
tests_green: res.testsGreen,
},
});
}
async function main() {
const cycle_id = `bot-${new Date().toISOString().replace(/[:.]/g, "-")}-${randomUUID().slice(0, 8)}`;
const started_at = new Date().toISOString();
log(`cycle ${cycle_id} starting`);
const finish = async (outcome: CycleOutcome, reason: string, extra: Partial<CycleResult> = {}): Promise<CycleResult> => {
const res: CycleResult = {
cycle_id,
started_at,
ended_at: new Date().toISOString(),
outcome,
reason,
gap: null,
proposal: null,
filesAdded: [],
filesUpdated: [],
filesNoop: [],
testsGreen: null,
testsOutput: "",
prUrl: null,
tokens_used: 0,
cloud_calls: 0,
...extra,
};
await persistResult(res);
await emit(res);
log(`→ ${outcome}: ${reason}`);
return res;
};
// 1. Pause.
if (await fileExists(PAUSE_FILE)) {
return finish("skipped_pause", `pause file at ${PAUSE_FILE}`);
}
// 2. Cost.
const cost = await readCost();
const bc = budgetCheck(cost);
if (!bc.ok) return finish("skipped_cost", bc.reason);
log(`cost: ${bc.reason}`);
// 3+4. Context + policy.shouldRunCycle.
const ctx = await gatherContext(cost);
const srq = policy.shouldRunCycle(ctx);
if (!srq.run) {
return finish(ctx.workingTreeDirty ? "skipped_dirty_tree" : "skipped_policy", srq.reason);
}
// 5. Gaps + policy.pickGap.
const gaps = await findGaps();
log(`found ${gaps.length} [bot-eligible] gap(s) in PRD`);
const gapPick = policy.pickGap(gaps);
if (!gapPick.gap) return finish("skipped_no_gap", gapPick.reason);
const gap = gapPick.gap;
log(`picked gap ${gap.id}: "${gap.prd_line.slice(0, 80)}"`);
// 6. KB lookup + Proposal.
// Bot cycles compound: prior cycles on this gap inform the current
// proposal. Summary is a compact block injected into the cloud prompt
// so the model can build on past successes and steer around past
// failures. Empty string when this is the first cycle on this gap.
const history = await loadHistory(gap.id, 5);
const stats = statsFor(history);
if (history.length > 0) {
log(`kb: ${stats.attempts} prior cycle(s) — ${stats.pr_opened} pr / ${stats.tests_failed} tests_failed / ${stats.proposal_rejected} rejected / ${stats.noop} noop`);
} else {
log(`kb: first cycle on this gap`);
}
const historySummary = summarizeHistory(history);
let proposal: Proposal;
try {
proposal = await generateProposal(gap, historySummary);
log(`proposal: ${proposal.summary} (${proposal.files.length} files, ~${proposal.estimated_loc} LOC, ${proposal.tokens_used} tokens)`);
} catch (e) {
await recordCost(1, 0);
return finish("model_failed", (e as Error).message, { gap, cloud_calls: 1 });
}
await recordCost(1, proposal.tokens_used);
// 7. Score.
const score = policy.scoreProposal(proposal);
if (!score.accept) {
return finish("proposal_rejected", score.reason, { gap, proposal, tokens_used: proposal.tokens_used, cloud_calls: 1 });
}
log(`proposal accepted: ${score.reason}`);
// 8. Branch.
const dayStamp = new Date().toISOString().slice(0, 10).replace(/-/g, "");
const branch = `bot/cycle-${dayStamp}-${gap.id}`;
// Branch must be fresh. If one already exists from a prior failed cycle
// for the same gap, bail — don't silently overwrite or reuse stale state.
const check = git(["rev-parse", "--verify", `refs/heads/${branch}`]);
if (check.code === 0) {
return finish("apply_failed",
`branch ${branch} already exists — delete it manually if you want a fresh cycle for this gap`,
{ gap, proposal, tokens_used: proposal.tokens_used, cloud_calls: 1 });
}
const co = git(["checkout", "-b", branch]);
if (co.code !== 0) {
return finish("apply_failed", `git checkout -b failed: ${co.stderr}`,
{ gap, proposal, tokens_used: proposal.tokens_used, cloud_calls: 1 });
}
log(`on branch ${branch}`);
// 9. Apply — Mem0 ADD/UPDATE/NOOP semantics.
const ap = await applyProposal(proposal);
if (ap.errors.length > 0) {
git(["checkout", "main"]);
git(["branch", "-D", branch]);
return finish("apply_failed", ap.errors.join("; "),
{ gap, proposal, tokens_used: proposal.tokens_used, cloud_calls: 1 });
}
const changed = [...ap.added, ...ap.updated];
log(`apply: ${ap.added.length} add, ${ap.updated.length} update, ${ap.noop.length} noop`);
// Short-circuit: if every proposed file was identical to what's on
// disk, there's nothing to test or PR. Clean up the branch and exit.
if (changed.length === 0) {
git(["checkout", "main"]);
git(["branch", "-D", branch]);
return finish("cycle_noop",
`all ${proposal.files.length} proposed file(s) were identical to current state`,
{ gap, proposal, filesNoop: ap.noop,
tokens_used: proposal.tokens_used, cloud_calls: 1 });
}
// 10. Test.
log(`running tests (this will take a while)...`);
const testRes = await runTests();
if (!testRes.green) {
git(["checkout", "."]);
git(["checkout", "main"]);
git(["branch", "-D", branch]);
return finish("tests_failed", "cargo or bun test failed", {
gap, proposal,
filesAdded: ap.added, filesUpdated: ap.updated, filesNoop: ap.noop,
testsGreen: false, testsOutput: testRes.output,
tokens_used: proposal.tokens_used, cloud_calls: 1,
});
}
log(`tests green`);
// 11. PR gate.
const partial: CycleResult = {
cycle_id, started_at,
ended_at: new Date().toISOString(),
outcome: "ok", reason: "",
gap, proposal,
filesAdded: ap.added, filesUpdated: ap.updated, filesNoop: ap.noop,
testsGreen: true, testsOutput: testRes.output,
prUrl: null, tokens_used: proposal.tokens_used, cloud_calls: 1,
};
const prGate = policy.shouldOpenPR(partial);
if (!prGate.open) {
git(["checkout", "main"]);
git(["branch", "-D", branch]);
return finish("pr_skipped_by_policy", prGate.reason, partial);
}
// 12. Commit + push.
git(["add", ...changed]);
const commitMsg = [
`bot: ${proposal.summary}`,
"",
proposal.rationale,
"",
`PRD gap: ${gap.prd_line}`,
`Cycle: ${cycle_id}`,
"",
`Co-Authored-By: ${proposal.model_used} (cloud) <bot@lakehouse.local>`,
].join("\n");
const ci = git(["commit", "-m", commitMsg]);
if (ci.code !== 0) {
return finish("pr_failed", `git commit failed: ${ci.stderr}`, partial);
}
const ps = git(["push", "-u", "origin", branch]);
if (ps.code !== 0) {
return finish("pr_failed", `git push failed: ${ps.stderr}`, partial);
}
// 13. Open PR.
try {
const pr = await openPr({
branch,
title: `bot: ${proposal.summary}`,
body: [
`**Gap** (PRD line ${gap.line_number}): ${gap.prd_line}`,
"",
`**Rationale**`,
proposal.rationale,
"",
`**Cycle**: \`${cycle_id}\``,
`**Model**: \`${proposal.model_used}\``,
`**Tokens**: ${proposal.tokens_used}`,
`**Added**: ${ap.added.map(f => `\`${f}\``).join(", ") || "_none_"}`,
`**Updated**: ${ap.updated.map(f => `\`${f}\``).join(", ") || "_none_"}`,
ap.noop.length > 0 ? `**NOOP (identical to current)**: ${ap.noop.map(f => `\`${f}\``).join(", ")}` : "",
].join("\n"),
});
return finish("ok", `PR #${pr.number} opened`, { ...partial, prUrl: pr.html_url });
} catch (e) {
return finish("pr_failed", (e as Error).message, partial);
}
}
main().catch(e => {
console.error("[bot] fatal:", e);
process.exit(1);
});
bot/kb.ts
+101
View File
@@ -0,0 +1,101 @@
// Bot-local knowledge base. Every finished cycle already persists a
// CycleResult to data/_bot/cycles/{id}.json — that IS the outcome log.
// KB here just reads that dir, filters to prior cycles on the same gap,
// and produces a short summary the cloud model can condition on.
//
// No separate jsonl, no new write path, no embedding calls. The bot's
// "memory" is the same primary artifact that the observer consumes.
//
// Future: embedding-based neighbor matching across gaps (cheap once
// sidecar is local), cross-pollination with scenario KB's
// pathway_recommendations. Not required for the feedback loop to work
// on a single gap — that's the floor we're building first.
import { readdir, readFile } from "node:fs/promises";
import { join } from "node:path";
import type { CycleResult } from "./types.ts";
const CYCLES_DIR = "/home/profit/lakehouse/data/_bot/cycles";
export interface HistoryEntry {
cycle_id: string;
ended_at: string;
outcome: string;
reason: string;
pr_url: string | null;
tests_green: boolean | null;
files_added: string[];
files_updated: string[];
tokens_used: number;
}
export async function loadHistory(gap_id: string, max: number = 5): Promise<HistoryEntry[]> {
let entries: string[] = [];
try {
entries = await readdir(CYCLES_DIR);
} catch {
return [];
}
const matches: HistoryEntry[] = [];
for (const e of entries) {
if (!e.endsWith(".json")) continue;
try {
const raw = await readFile(join(CYCLES_DIR, e), "utf8");
const r = JSON.parse(raw) as CycleResult;
if (r.gap?.id !== gap_id) continue;
matches.push({
cycle_id: r.cycle_id,
ended_at: r.ended_at,
outcome: r.outcome,
reason: r.reason,
pr_url: r.prUrl,
tests_green: r.testsGreen,
files_added: r.filesAdded ?? [],
files_updated: r.filesUpdated ?? [],
tokens_used: r.tokens_used,
});
} catch {
// Skip unreadable / malformed cycle files. Don't fail the current
// cycle because an old one is corrupt.
}
}
matches.sort((a, b) => b.ended_at.localeCompare(a.ended_at));
return matches.slice(0, max);
}
// Compact prompt-ready summary. Empty string when there's no history —
// caller can skip the "prior attempts" block entirely.
export function summarizeHistory(h: HistoryEntry[]): string {
if (h.length === 0) return "";
const lines = h.map(e => {
const when = e.ended_at.slice(0, 16).replace("T", " ");
const files = [...e.files_added, ...e.files_updated];
const filesStr = files.length > 0 ? ` touched: ${files.join(", ")}` : "";
const prStr = e.pr_url ? ` PR: ${e.pr_url}` : "";
return `- ${when} UTC — ${e.outcome}${prStr}${filesStr}\n reason: ${e.reason}`;
});
return [
`Prior attempts on this gap (${h.length} most recent):`,
...lines,
"",
"Learn from these: build on what worked, avoid paths that failed.",
].join("\n");
}
// Aggregate stats for telemetry — lets the bot expose "% of cycles on
// this gap that landed a PR" without re-parsing the raw history.
export function statsFor(h: HistoryEntry[]): {
attempts: number;
pr_opened: number;
tests_failed: number;
proposal_rejected: number;
noop: number;
} {
return {
attempts: h.length,
pr_opened: h.filter(e => e.outcome === "ok").length,
tests_failed: h.filter(e => e.outcome === "tests_failed").length,
proposal_rejected: h.filter(e => e.outcome === "proposal_rejected").length,
noop: h.filter(e => e.outcome === "cycle_noop").length,
};
}
bot/observer.ts
+34
View File
@@ -0,0 +1,34 @@
// POST bot cycle outcomes to the observer on :3800 so they accumulate
// in the KB alongside scenario + autotune events. Non-fatal on failure:
// the cycle result is also written to disk so observability isn't a
// single point of failure.
const OBSERVER_URL = process.env.LH_OBSERVER_URL ?? "http://localhost:3800";
export interface BotObserverEvent {
source: "bot";
cycle_id: string;
sig_hash: string; // stable hash of the gap + proposal for dedup
event_kind: string; // cycle outcome (ok, tests_failed, ...)
ok: boolean;
staffer_id?: string; // the model used, e.g. "gpt-oss:120b"
turns?: number; // number of cloud calls this cycle
duration_ms?: number;
extra?: Record<string, any>;
}
export async function postEvent(ev: BotObserverEvent): Promise<void> {
try {
const r = await fetch(`${OBSERVER_URL}/event`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(ev),
signal: AbortSignal.timeout(3000),
});
if (!r.ok) {
console.error(`[bot/observer] ${r.status}: ${await r.text()}`);
}
} catch (e) {
console.error(`[bot/observer] POST failed: ${(e as Error).message}`);
}
}
bot/policy.ts
+70
View File
@@ -0,0 +1,70 @@
// ═══════════════════════════════════════════════════════════════════
// YOU WRITE THIS FILE. These four functions shape the bot's entire
// behavior. Defaults below are safe and intentionally conservative —
// redline to taste. Each function is ~5-10 lines.
// ═══════════════════════════════════════════════════════════════════
import type { CycleContext, CycleResult, Gap, Proposal } from "./types.ts";
// Should this cycle even fire right now?
// Called AFTER the pause-file check + cost-cap check have already passed.
// Use this for judgment calls (cooldown, shared-resource conflicts).
export function shouldRunCycle(ctx: CycleContext): { run: boolean; reason: string } {
if (ctx.workingTreeDirty) {
return { run: false, reason: "dirty working tree — commit or stash your work first" };
}
if (ctx.lastCycleAt) {
const minsAgo = (Date.now() - new Date(ctx.lastCycleAt).getTime()) / 60000;
if (minsAgo < 10) return { run: false, reason: `last cycle ${minsAgo.toFixed(1)}min ago (<10min cooldown)` };
}
if (ctx.autotuneBusy) {
return { run: false, reason: "autotune agent is running — avoid Ollama contention" };
}
return { run: true, reason: "ready" };
}
// Which [bot-eligible] gap from the PRD does this cycle attack?
// Default: first in document order. Alternatives: random for exploration,
// smallest-context, tag-priority (e.g. [bot-eligible:high] wins).
export function pickGap(gaps: Gap[]): { gap: Gap | null; reason: string } {
if (gaps.length === 0) return { gap: null, reason: "no [bot-eligible] gaps in PRD" };
return { gap: gaps[0], reason: "first eligible gap in document order" };
}
// Cheap gate BEFORE burning test cycles. Reject if proposal smells wrong.
// This catches runaway cloud proposals before we apply+test them.
export function scoreProposal(p: Proposal): { accept: boolean; reason: string } {
if (p.files.length === 0) return { accept: false, reason: "empty proposal" };
if (p.files.length > 5) return { accept: false, reason: `>5 files (${p.files.length})` };
if (p.estimated_loc > 200) return { accept: false, reason: `>200 LOC (${p.estimated_loc})` };
// Paths the bot is NEVER allowed to touch. Extend cautiously.
const forbidden = [
".git/", "secrets", "lakehouse.toml",
"docs/ADR-", "docs/DECISIONS.md", "docs/PRD.md",
"/etc/", "/root/", "Cargo.lock",
];
for (const f of p.files) {
const hit = forbidden.find(prefix => f.path.includes(prefix));
if (hit) return { accept: false, reason: `touches forbidden path (${hit}): ${f.path}` };
}
return { accept: true, reason: "passes size + path filters" };
}
// Tests are green and changes applied. Do we actually open a PR?
// Mem0-aligned: the cycle already short-circuited with `cycle_noop` if
// every file was identical to current state, so by the time we're here
// at least one file was actually added or updated.
//
// Stricter options below — uncomment to require a test file touched, etc.
export function shouldOpenPR(c: CycleResult): { open: boolean; reason: string } {
if (!c.testsGreen) return { open: false, reason: "tests red" };
const realChanges = c.filesAdded.length + c.filesUpdated.length;
if (realChanges === 0) return { open: false, reason: "nothing actually changed on disk" };
// Stricter option — require at least one test file in the diff:
// const hasTest = [...c.filesAdded, ...c.filesUpdated].some(f => /test|_tests/i.test(f));
// if (!hasTest) return { open: false, reason: "no test file touched" };
return { open: true, reason: `green + ${c.filesAdded.length} add / ${c.filesUpdated.length} update` };
}
bot/pr.ts
+58
View File
@@ -0,0 +1,58 @@
// Gitea PR open. Reads PAT from ~/.git-credentials (set up by the
// credential-helper flow). All PRs open as DRAFT so J reviews before
// merge — never auto-merge.
import { readFile } from "node:fs/promises";
const GITEA_HOST = "https://git.agentview.dev";
const REPO_OWNER = "profit";
const REPO_NAME = "lakehouse";
const CRED_FILE = "/home/profit/.git-credentials";
async function readPat(): Promise<string> {
const raw = await readFile(CRED_FILE, "utf8");
for (const line of raw.split("\n")) {
const m = line.match(/^https:\/\/[^:]+:([^@]+)@git\.agentview\.dev/);
if (m) return m[1];
}
throw new Error(`no Gitea PAT found in ${CRED_FILE}`);
}
export interface OpenPrInput {
branch: string; // head branch (just pushed)
base?: string; // default "main"
title: string;
body: string;
}
export interface OpenPrResult {
number: number;
html_url: string;
}
export async function openPr(input: OpenPrInput): Promise<OpenPrResult> {
const pat = await readPat();
const url = `${GITEA_HOST}/api/v1/repos/${REPO_OWNER}/${REPO_NAME}/pulls`;
const payload = {
title: input.title,
body: input.body,
head: input.branch,
base: input.base ?? "main",
// Gitea uses "draft" flag — always on for bot PRs.
draft: true,
};
const r = await fetch(url, {
method: "POST",
headers: {
"content-type": "application/json",
"Authorization": `token ${pat}`,
},
body: JSON.stringify(payload),
signal: AbortSignal.timeout(30000),
});
if (!r.ok) {
throw new Error(`Gitea ${r.status}: ${await r.text()}`);
}
const j = await r.json() as any;
return { number: j.number, html_url: j.html_url };
}
bot/propose.ts
+141
View File
@@ -0,0 +1,141 @@
// Gap detection + cloud proposal.
//
// Gap detection: scan docs/PRD.md for lines tagged [bot-eligible].
// Each match becomes a Gap with surrounding context.
//
// Proposal: one-shot call to the T3 cloud model via the Python sidecar's
// /generate endpoint. Asks for a structured JSON response with file
// contents. Truncation-resistant via Phase 21's generate_continuable —
// for now we pass max_tokens high and rely on the model completing in
// one pass; swap to the Rust continuation wrapper if we see truncation.
import { readFile } from "node:fs/promises";
import { createHash } from "node:crypto";
import type { Gap, Proposal } from "./types.ts";
const SIDECAR_URL = process.env.LH_SIDECAR_URL ?? "http://localhost:3200";
const REPO_ROOT = "/home/profit/lakehouse";
const PRD_PATH = `${REPO_ROOT}/docs/PRD.md`;
const CLOUD_MODEL = process.env.LH_BOT_MODEL ?? "gpt-oss:120b";
const MAX_TOKENS = 6000;
export async function findGaps(): Promise<Gap[]> {
const prd = await readFile(PRD_PATH, "utf8");
const lines = prd.split("\n");
const gaps: Gap[] = [];
for (let i = 0; i < lines.length; i++) {
if (!lines[i].includes("[bot-eligible]")) continue;
const contextLines = lines.slice(i, Math.min(i + 6, lines.length)).join("\n");
const id = createHash("sha256").update(lines[i]).digest("hex").slice(0, 12);
gaps.push({
id,
prd_line: lines[i].trim(),
context: contextLines,
source_file: "docs/PRD.md",
line_number: i + 1,
});
}
return gaps;
}
const SYSTEM_PROMPT = `You are an assistant that proposes small, testable code changes to the Lakehouse repo.
The Lakehouse is a Rust-first data platform with 13 crates + Bun/TypeScript test harness.
You will be given one PRD gap tagged [bot-eligible] and must respond with a STRICT JSON object — no prose.
Rules:
- Response MUST be a single JSON object, no markdown fences, no commentary.
- Change MUST be small: <200 lines total, ≤5 files.
- Include at least one test file (new or modified) that proves the change.
- NEVER touch .git/, secrets, lakehouse.toml, docs/ADR-*, docs/DECISIONS.md, docs/PRD.md, /etc/, /root/, Cargo.lock.
- Paths MUST be repo-relative (no leading /).
- Whole-file contents only — no patches, no diffs.
Response shape:
{
"summary": "one line",
"rationale": "why this addresses the gap",
"files": [ { "path": "crates/foo/src/bar.rs", "content": "<full file>", "is_new": false } ],
"estimated_loc": 42
}`;
export async function generateProposal(gap: Gap, historySummary: string = ""): Promise<Proposal> {
const sections = [
`PRD gap (line ${gap.line_number}):`,
"```",
gap.context,
"```",
"",
];
if (historySummary) {
sections.push(historySummary, "");
}
sections.push("Propose a small change that addresses this gap. Respond with the JSON object only.");
const userPrompt = sections.join("\n");
const r = await fetch(`${SIDECAR_URL}/generate`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
model: CLOUD_MODEL,
system: SYSTEM_PROMPT,
prompt: userPrompt,
temperature: 0.2,
max_tokens: MAX_TOKENS,
think: false,
}),
signal: AbortSignal.timeout(180000), // cloud T3 can be slow — 3 min
});
if (!r.ok) {
throw new Error(`sidecar ${r.status}: ${await r.text()}`);
}
const j = await r.json() as any;
const raw: string = j.text ?? j.response ?? "";
const usage = j.usage ?? {};
const tokens = (usage.prompt_tokens ?? 0) + (usage.completion_tokens ?? 0);
const parsed = extractJson(raw);
if (!parsed || typeof parsed !== "object") {
throw new Error(`model returned no JSON object. Raw head: ${raw.slice(0, 300)}`);
}
if (!Array.isArray(parsed.files)) {
throw new Error(`proposal.files not an array: ${JSON.stringify(parsed).slice(0, 200)}`);
}
return {
summary: String(parsed.summary ?? "").trim(),
rationale: String(parsed.rationale ?? "").trim(),
files: parsed.files.map((f: any) => ({
path: String(f.path ?? ""),
content: String(f.content ?? ""),
is_new: Boolean(f.is_new),
})).filter((f: any) => f.path && f.content !== undefined),
estimated_loc: Number(parsed.estimated_loc ?? 0),
model_used: CLOUD_MODEL,
tokens_used: tokens,
};
}
// Find the first balanced JSON object in the string. Tolerates leading
// "```json" fences even though we asked the model not to emit them.
function extractJson(text: string): any | null {
const cleaned = text.replace(/^```(?:json)?\s*/i, "").replace(/```\s*$/i, "").trim();
let depth = 0;
let start = -1;
for (let i = 0; i < cleaned.length; i++) {
const c = cleaned[i];
if (c === "{") {
if (depth === 0) start = i;
depth++;
} else if (c === "}") {
depth--;
if (depth === 0 && start >= 0) {
try {
return JSON.parse(cleaned.slice(start, i + 1));
} catch {
start = -1;
}
}
}
}
return null;
}
bot/test.ts
+46
View File
@@ -0,0 +1,46 @@
// Run the test gate: `cargo test --workspace` + `bun test`.
// Returns { green, output } — output is last ~4KB of combined stdout/stderr.
import { spawn } from "node:child_process";
const REPO_ROOT = "/home/profit/lakehouse";
const TEST_TIMEOUT_MS = 15 * 60 * 1000; // 15 min cargo + 2 min bun, generous
export async function runTests(): Promise<{ green: boolean; output: string }> {
const cargoOut = await runCmd("cargo", ["test", "--workspace", "--quiet", "--", "--test-threads=1"], REPO_ROOT, TEST_TIMEOUT_MS);
if (cargoOut.code !== 0) {
return { green: false, output: tail(`cargo test (${cargoOut.code}):\n${cargoOut.combined}`) };
}
const bunOut = await runCmd("bun", ["test", "tests/multi-agent"], REPO_ROOT, 120000);
const bunGreen = bunOut.code === 0;
const combined = [
`cargo test: OK`,
`bun test (${bunOut.code}):`,
bunOut.combined,
].join("\n");
return { green: bunGreen, output: tail(combined) };
}
function tail(s: string, n = 4096): string {
return s.length > n ? "…" + s.slice(-n) : s;
}
function runCmd(
cmd: string,
args: string[],
cwd: string,
timeoutMs: number,
): Promise<{ code: number; combined: string }> {
return new Promise(resolveP => {
const child = spawn(cmd, args, { cwd, env: { ...process.env } });
let combined = "";
child.stdout.on("data", d => { combined += d.toString(); });
child.stderr.on("data", d => { combined += d.toString(); });
const timer = setTimeout(() => child.kill("SIGKILL"), timeoutMs);
child.on("close", code => {
clearTimeout(timer);
resolveP({ code: code ?? -1, combined });
});
});
}
bot/types.ts
+88
View File
@@ -0,0 +1,88 @@
// Shared types for the PR-bot. Small and explicit — the bot's behavior
// should be readable from these shapes alone.
export interface Gap {
id: string; // hash of prd_line — stable key for dedup
prd_line: string; // the [bot-eligible]-tagged line, verbatim
context: string; // surrounding context from the PRD (next ~5 lines)
source_file: string; // e.g. "docs/PRD.md"
line_number: number;
}
export interface ProposalFile {
path: string; // repo-relative (no leading slash)
content: string; // full file content — the bot writes whole files, not patches
is_new: boolean; // true = create, false = overwrite existing
}
export interface Proposal {
summary: string; // one-line
rationale: string; // why this change addresses the gap
files: ProposalFile[];
estimated_loc: number; // total added+changed lines across all files
model_used: string;
tokens_used: number;
}
export interface CycleContext {
startedAt: string; // ISO
dailyCallsUsed: number;
dailyCallsBudget: number;
dailyTokensUsed: number;
dailyTokensBudget: number;
lastCycleAt: string | null;
lastCycleGapId: string | null;
autotuneBusy: boolean;
workingTreeDirty: boolean;
}
export type CycleOutcome =
| "ok" // PR opened
| "skipped_pause"
| "skipped_cost"
| "skipped_policy"
| "skipped_no_gap"
| "skipped_dirty_tree"
| "proposal_rejected"
| "apply_failed"
| "cycle_noop" // proposal applied but every file was identical to what's on disk
| "tests_failed"
| "pr_skipped_by_policy"
| "pr_failed"
| "model_failed";
// Mem0-aligned apply outcomes. Three shapes instead of the binary
// "written" / "errored". NOOP is a first-class outcome — identical
// content shouldn't waste test cycles or open an empty PR.
export type ApplyMode = "add" | "update" | "noop";
export interface ApplyOutcome {
added: string[];
updated: string[];
noop: string[];
errors: string[];
}
export interface CycleResult {
cycle_id: string;
started_at: string;
ended_at: string;
outcome: CycleOutcome;
reason: string;
gap: Gap | null;
proposal: Proposal | null;
filesAdded: string[];
filesUpdated: string[];
filesNoop: string[];
testsGreen: boolean | null; // null = not run
testsOutput: string;
prUrl: string | null;
tokens_used: number;
cloud_calls: number;
}
export interface CostState {
date: string; // YYYY-MM-DD UTC
calls: number;
tokens: number;
}
bun.lock
+18
View File
@@ -0,0 +1,18 @@
{
"lockfileVersion": 1,
"configVersion": 1,
"workspaces": {
"": {
"dependencies": {
"langfuse": "^3.38.20",
},
},
},
"packages": {
"langfuse": ["langfuse@3.38.20", "", { "dependencies": { "langfuse-core": "^3.38.20" } }, "sha512-MAmBAASSzJtmK1O9HQegA1mFsQhT8Yf+OJRGvE7FXkyv3g/eiBE0glLD0Ohg3pkxhoPdggM5SejK7ue9ctlaMA=="],
"langfuse-core": ["langfuse-core@3.38.20", "", { "dependencies": { "mustache": "^4.2.0" } }, "sha512-zBKVmQN/1oT5VWZUBYlWzvokIlkC/6mnpgr/2atMyTeAm+jR3ia7w2iJMjlrF5/oG8ukO1s8+LDRCzJpF1QeEA=="],
"mustache": ["mustache@4.2.0", "", { "bin": { "mustache": "bin/mustache" } }, "sha512-71ippSywq5Yb7/tVYyGbkBggbU8H3u5Rz56fH60jGFgr8uHwxs+aSKeqmluIVzM0m0kB7xQjKS6qPfd0b2ZoqQ=="],
}
}
config/models.json
+193
View File
@@ -0,0 +1,193 @@
{
"_description": "Lakehouse model matrix — authoritative routing for all agent tiers. Local models do the heavy lifting; cloud models are consulted sparingly for overview, strategic, and gatekeeper decisions. Read by tests/multi-agent/scenario.ts and mcp-server/index.ts.",
"version": 1,
"updated": "2026-04-21",
"providers": {
"ollama_local": {
"base_url": "http://localhost:11434",
"key_env": null
},
"ollama_cloud": {
"base_url": "https://ollama.com",
"key_env": "OLLAMA_CLOUD_KEY",
"key_source": "/root/llm_team_config.json → providers.ollama_cloud.api_key",
"rate_budget": {
"calls_per_hour": 200,
"calls_per_day": 2000,
"notes": "Paid tier — generous. Policy: keep overview calls ≤ 3/scenario, strategic ≤ 10/day, gatekeeper ≤ 5/day."
}
}
},
"tiers": {
"t1_hot": {
"purpose": "Per tool call — SQL generation, hybrid_search, sql(). Runs 50-200 times per scenario. Latency-sensitive.",
"kind": "local_fast",
"primary": { "model": "qwen3.5:latest", "provider": "ollama_local", "context_window": 262144 },
"fallback": { "model": "qwen2.5:latest", "provider": "ollama_local", "context_window": 32768 },
"max_tokens": 1000,
"temperature": 0.3,
"never_route_cloud": true,
"context_budget": {
"system_prompt_cap": 4000,
"prior_context_cap": 6000,
"tool_results_cap": 8000,
"safety_margin": 2000,
"overflow_policy": "summarize_oldest_tool_results_via_t3"
},
"rationale": "qwen3.5:latest is a 9.7B thinking model with 262K context that emits clean JSON (verified 2026-04-21 after mistral A/B wipeout). Thinking budget requires max_tokens ≥ 400; we hold 1000 to keep propose_done payloads intact. qwen2.5 stays as fallback."
},
"t2_review": {
"purpose": "Per step consensus — executor ↔ reviewer loop critique. 5-14 calls per event.",
"kind": "local_balanced",
"primary": { "model": "qwen3:latest", "provider": "ollama_local", "context_window": 40960 },
"fallback": { "model": "qwen2.5:latest", "provider": "ollama_local", "context_window": 32768 },
"max_tokens": 800,
"temperature": 0.3,
"never_route_cloud": true,
"context_budget": {
"system_prompt_cap": 2000,
"recent_turns_cap": 4000,
"safety_margin": 1000
},
"rationale": "Reviewer only needs to detect schema violations and drift — a 7B model is sufficient."
},
"t3_overview": {
"purpose": "Mid-day checkpoint after every misplacement + every Nth event, and cross-day lesson. 1-3 calls per scenario.",
"kind": "thinking_cloud",
"primary": { "model": "gpt-oss:120b", "provider": "ollama_cloud", "context_window": 131072 },
"local_fallback": { "model": "gpt-oss:20b", "provider": "ollama_local", "context_window": 131072 },
"max_tokens": 900,
"temperature": 0.2,
"cloud_budget_per_scenario": 5,
"env_flag": "LH_OVERVIEW_CLOUD=1",
"context_budget": {
"event_digest_cap": 30000,
"checkpoint_cap": 8000,
"lesson_corpus_cap": 40000,
"safety_margin": 8000,
"overflow_policy": "chunk_lessons_via_cosine_topk"
},
"rationale": "Same prompt family as local 20b (gpt-oss series) — prompts port directly. 120b is faster via cloud than 20b local in practice, and lessons are noticeably more specific."
},
"t4_strategic": {
"purpose": "Daily playbook board re-ranking, weekly gap audit, pattern discovery across accumulated playbooks. 1-10 calls per day.",
"kind": "thinking_cloud_large",
"primary": { "model": "kimi-k2.6", "provider": "ollama_cloud", "context_window": 200000 },
"secondary": { "model": "qwen3.5:397b", "provider": "ollama_cloud", "context_window": 131072 },
"fallback": { "model": "glm-4.7", "provider": "ollama_cloud", "context_window": 131072 },
"local_fallback": { "model": "gpt-oss:20b", "provider": "ollama_local", "context_window": 131072 },
"max_tokens": 2000,
"temperature": 0.2,
"cloud_budget_per_day": 10,
"context_budget": {
"playbook_corpus_cap": 80000,
"pattern_history_cap": 20000,
"safety_margin": 16000,
"overflow_policy": "two_pass_map_reduce"
},
"rationale": "J named qwen3.5 specifically. GLM-4.7 is a promising alternate for debate phase. Runs after all scenarios complete for the day."
},
"t5_gatekeeper": {
"purpose": "MUST route here: architecture changes, new client onboarding, schema migrations, playbook retirements, index rebuilds, autotune config changes.",
"kind": "thinking_cloud_deepest",
"primary": { "model": "kimi-k2-thinking", "provider": "ollama_cloud", "context_window": 200000 },
"fallback": { "model": "deepseek-v3.1:671b", "provider": "ollama_cloud", "context_window": 131072 },
"secondary_fallback": { "model": "qwen3.5:397b", "provider": "ollama_cloud", "context_window": 131072 },
"local_fallback": { "model": "gpt-oss:20b", "provider": "ollama_local", "context_window": 131072 },
"max_tokens": 4000,
"temperature": 0.1,
"cloud_budget_per_day": 5,
"audit_log": true,
"context_budget": {
"decision_doc_cap": 50000,
"evidence_bundle_cap": 100000,
"prior_gatekeeper_decisions_cap": 20000,
"safety_margin": 20000,
"overflow_policy": "escalate_to_kimi_k2_1t_or_split_decision"
},
"rationale": "Highest-stakes decisions — reasoning depth matters more than latency. Audit log so J can always see what the gatekeeper was asked and what it answered. No human approval required today; escalate later if mis-decisions show up."
}
},
"embeddings": {
"_description": "Vector embedding models for hybrid search + playbook_memory similarity. Always local — embedding calls are too high-volume for cloud. qwen3-embedding is the new primary (2026-04-21); nomic-embed-text-v2-moe is the MoE alternative for corpus-scale embedding; original nomic-embed-text stays as legacy fallback while vectord indexes are being re-embedded.",
"primary": { "model": "qwen3-embedding", "provider": "ollama_local", "dim": 1024, "context_window": 32768 },
"alternate": { "model": "nomic-embed-text-v2-moe", "provider": "ollama_local", "dim": 768, "context_window": 2048, "notes": "MoE variant — better for long documents, slower per-call, roughly matches nomic-embed-text on short text." },
"legacy": { "model": "nomic-embed-text:latest", "provider": "ollama_local", "dim": 768, "context_window": 8192, "notes": "Used by existing vectord indexes. Do NOT switch live until indexes are rebuilt, or hybrid search recall will crater." },
"migration_plan": "Re-embed workers_500k (and all production indexes) with qwen3-embedding, keep the old index under a _v1 suffix for rollback, flip the profile pointer only after recall benchmarks match or exceed legacy. Touches vectord::agent + autotune."
},
"context_management": {
"_description": "Rule zero: NEVER call a model with more tokens than its context_window minus safety_margin. Every call goes through the budget checker first. If over budget → chunk, summarize, or escalate. This is the stability floor.",
"token_estimator": {
"method": "chars_div_4",
"note": "Rough, biased safe by ~15%. For production, swap to tiktoken or the provider's tokenizer endpoint."
},
"overflow_policies": {
"summarize_oldest_tool_results_via_t3": {
"when": "T1 conversation history + tool results exceed context_budget.tool_results_cap",
"how": "Send oldest N tool results to T3 with prompt 'summarize these into 500 tokens that preserve what the executor needs to know'; replace them with the summary in the running conversation."
},
"chunk_lessons_via_cosine_topk": {
"when": "lesson corpus in data/_playbook_lessons/*.json exceeds lesson_corpus_cap",
"how": "Embed the current scenario spec, cosine-rank all lessons, take top-K until budget exhausted. Fall back to date-sorted if embeddings unavailable."
},
"two_pass_map_reduce": {
"when": "T4 playbook corpus exceeds playbook_corpus_cap",
"how": "Pass 1: chunk playbooks into ≤30K token shards, run primary model on each shard to emit 'shard summary'. Pass 2: feed all summaries to primary model for global synthesis. Logged as two audit entries."
},
"escalate_to_kimi_k2_1t_or_split_decision": {
"when": "T5 decision evidence exceeds decision_doc_cap + evidence_bundle_cap",
"how": "Prefer kimi-k2:1t which has 1M context. If still over, split decision into sub-decisions (e.g. 'retire playbooks by city' instead of 'retire playbooks globally') and loop."
}
},
"chunking_cache": {
"_description": "Precomputed shards of the playbook corpus, indexed by (corpus_version, shard_id). Avoids re-chunking on every T4 run.",
"location": "data/_chunk_cache/",
"invalidation": "Key includes corpus hash. When playbook_memory changes, the hash changes, the cache misses, and chunks regenerate.",
"implementation_status": "SPEC — next sprint."
},
"implementation_status": "context_window + context_budget fields WIRED in config. Enforcement helper NOT yet wired in agent.ts. Next: add estimateTokens() + budgetCheck() helpers, route all generate() calls through them, emit a metric when overflow policy fires."
},
"experimental_rotation": {
"enabled": false,
"purpose": "Sample newer models on a schedule to collect comparison data without rate-limit risk.",
"candidates": [
{ "model": "minimax-m2.7", "notes": "Newer minimax; unknown output stability" },
{ "model": "glm-5", "notes": "GLM next-gen; larger context" },
{ "model": "glm-5.1", "notes": "Incremental on GLM-5" },
{ "model": "qwen3-next:80b", "notes": "Qwen's experimental successor; smaller than 3.5" },
{ "model": "qwen3-coder-next", "notes": "Coder-optimized — good for SQL gen T1 experiments" },
{ "model": "deepseek-v3.2", "notes": "Smaller deepseek; reasoning/coding" },
{ "model": "nemotron-3-super", "notes": "NVIDIA 230B; general" },
{ "model": "cogito-2.1:671b", "notes": "671B general" },
{ "model": "mistral-large-3:675b", "notes": "Mistral's flagship; good T3 candidate" }
],
"rotation": "weekly",
"sample_rate": 0.1,
"apply_to_tier": "t4_strategic",
"notes": "When enabled, T4 routes 10% of calls to a rotating experimental model. Log comparison in /data/_model_eval/ — if the experimental consistently beats primary across 3 rotations, promote it to primary."
},
"playbook_versioning": {
"enabled": true,
"purpose": "A playbook can work, then break when architecture changes. Versioning lets us pin which change retired which playbook.",
"dataset": "playbook_memory",
"schema_additions": {
"version": "integer — auto-increment per operation",
"parent_id": "string — previous version entry_id for same operation (null for v1)",
"retired_at": "timestamp — set when success_rate drops or architecture changes",
"retirement_reason": "string — e.g. 'schema_migration:workers_500k 2026-05-03'",
"architecture_snapshot": "object — crate versions, index name, schema fingerprint at seed time"
},
"retire_triggers": [
"success_rate < 0.3 over last 20 citations",
"schema_fingerprint mismatch detected at retrieval time",
"architecture change event emitted by ingestd/vectord",
"T5 gatekeeper explicitly retires via /vectors/playbook_memory/retire"
],
"read_back_policy": "retrieval returns only non-retired versions. History endpoint /vectors/playbook_memory/history/{operation} returns the full chain.",
"ui_surface": "mcp-server to render a diff view: side-by-side of playbook versions with a timeline of what changed and when.",
"implementation_status": "SPEC — not yet wired. Target: next sprint. Touches gateway + catalogd + mcp-server."
},
"matrix_index_hybrid_search_note": "Phase 22 candidate: elevate the hybrid_search T1 tool to consult T3 when a pool returns <3 matches OR when the same (role, city) combo has failed N times in 24h. Consult result is a reformulated sql_filter the executor retries with. Keeps T1 fast on the happy path, escalates to T3 only on low-recall signals."
}
crates/aibridge/src/client.rs
+9 -1
View File
@@ -25,7 +25,7 @@ pub struct EmbedResponse {
pub dimensions: usize, pub dimensions: usize,
} }
#[derive(Serialize, Deserialize)] #[derive(Clone, Serialize, Deserialize)]
pub struct GenerateRequest { pub struct GenerateRequest {
pub prompt: String, pub prompt: String,
#[serde(skip_serializing_if = "Option::is_none")] #[serde(skip_serializing_if = "Option::is_none")]
@@ -36,6 +36,14 @@ pub struct GenerateRequest {
pub temperature: Option<f64>, pub temperature: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")] #[serde(skip_serializing_if = "Option::is_none")]
pub max_tokens: Option<u32>, pub max_tokens: Option<u32>,
/// Phase 21 — per-call opt-out of hidden reasoning. Thinking models
/// (qwen3.5, gpt-oss, etc) burn tokens on reasoning before the
/// visible response starts; setting this to `false` on hot-path
/// JSON emitters avoids empty returns when the budget is tight.
/// Sidecar forwards this to Ollama's `think` parameter; if the
/// sidecar drops an unknown field the request still succeeds.
#[serde(skip_serializing_if = "Option::is_none")]
pub think: Option<bool>,
} }
#[derive(Deserialize, Serialize, Clone)] #[derive(Deserialize, Serialize, Clone)]
crates/aibridge/src/context.rs
+194
View File
@@ -0,0 +1,194 @@
//! Phase 21 — context-budget accounting for model calls.
//!
//! Ports `assertContextBudget` + `estimateTokens` + `CONTEXT_WINDOWS`
//! from `tests/multi-agent/agent.ts` so Rust-side callers (gateway
//! tool surfaces, future Rust agents) get the same loud-fail behavior
//! on window overflow instead of silent truncation.
//!
//! The token estimator is deliberately the same chars/4 heuristic as
//! the TS side. It's biased ~15% safe — pessimistic on English, correct
//! within a factor of 2 on code. Swap to a provider tokenizer only when
//! the estimator drives a decision (we're nowhere near that yet).
use std::collections::HashMap;
use std::sync::OnceLock;
/// Rough token count. `chars / 4` ceiling. See module docs for why
/// this heuristic is sufficient.
pub fn estimate_tokens(text: &str) -> usize {
(text.chars().count() + 3) / 4
}
/// Phase 21 — per-model context windows, mirroring the TS table in
/// `tests/multi-agent/agent.ts`. Anchored on each model's documented
/// max; unknown models fall back to `DEFAULT_CONTEXT_WINDOW`.
pub const DEFAULT_CONTEXT_WINDOW: usize = 32_768;
pub const DEFAULT_SAFETY_MARGIN: usize = 2_000;
pub const DEFAULT_MAX_TOKENS: usize = 800;
fn known_windows() -> &'static HashMap<&'static str, usize> {
static TABLE: OnceLock<HashMap<&'static str, usize>> = OnceLock::new();
TABLE.get_or_init(|| {
let mut m = HashMap::new();
m.insert("mistral:latest", 32_768);
m.insert("qwen2.5:latest", 32_768);
m.insert("qwen3:latest", 40_960);
m.insert("qwen3.5:latest", 262_144);
m.insert("qwen3-embedding", 32_768);
m.insert("nomic-embed-text-v2-moe", 2_048);
m.insert("gpt-oss:20b", 131_072);
m.insert("gpt-oss:120b", 131_072);
m.insert("qwen3.5:397b", 131_072);
m.insert("kimi-k2-thinking", 200_000);
m.insert("kimi-k2.6", 200_000);
m.insert("kimi-k2:1t", 1_048_576);
m.insert("deepseek-v3.1:671b", 131_072);
m.insert("glm-4.7", 131_072);
m
})
}
pub fn context_window_for(model: &str) -> usize {
known_windows().get(model).copied().unwrap_or(DEFAULT_CONTEXT_WINDOW)
}
/// Result of a budget check — exposes the numbers so callers can log
/// how much headroom remains without re-running the estimator.
#[derive(Debug, Clone, Copy)]
pub struct BudgetCheck {
pub estimated: usize,
pub window: usize,
pub remaining: i64,
}
/// Inputs to `assert_context_budget`. `bypass` exists for call sites
/// that handle their own overflow (continuation's second pass already
/// counted the partial; T5 gatekeeper prompts have a separate policy).
#[derive(Debug, Clone, Default)]
pub struct BudgetOpts<'a> {
pub system: Option<&'a str>,
pub max_tokens: Option<usize>,
pub safety_margin: Option<usize>,
pub bypass: bool,
}
/// Phase 21's loud-fail primitive. Returns a `BudgetCheck` on success
/// and the same struct plus over-by count on failure. The whole point
/// is to stop silent truncation — callers that expect overflow should
/// chunk BEFORE calling or set `bypass: true`.
pub fn assert_context_budget(
model: &str,
prompt: &str,
opts: BudgetOpts,
) -> Result<BudgetCheck, (BudgetCheck, usize)> {
let window = context_window_for(model);
let safety = opts.safety_margin.unwrap_or(DEFAULT_SAFETY_MARGIN);
let max_tokens = opts.max_tokens.unwrap_or(DEFAULT_MAX_TOKENS);
let sys_tokens = opts.system.map(estimate_tokens).unwrap_or(0);
let estimated = estimate_tokens(prompt) + sys_tokens + max_tokens;
let remaining = window as i64 - estimated as i64 - safety as i64;
let check = BudgetCheck { estimated, window, remaining };
if remaining < 0 && !opts.bypass {
return Err((check, (-remaining) as usize));
}
Ok(check)
}
/// Convenience — format an overflow error the same way the TS side
/// does. Exposed so downstream crates render consistent messages.
pub fn overflow_message(model: &str, check: &BudgetCheck, over_by: usize, safety: usize) -> String {
format!(
"context overflow: model={} est={}t window={}t safety={}t over={}t. \
Chunk the prompt (see config/models.json overflow_policies) or set \
bypass:true if you know the risk.",
model, check.estimated, check.window, safety, over_by,
)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn estimate_tokens_ceiling_divides_by_four() {
assert_eq!(estimate_tokens(""), 0);
assert_eq!(estimate_tokens("abc"), 1); // 3 → ceil(3/4) = 1
assert_eq!(estimate_tokens("abcd"), 1); // 4 → ceil(4/4) = 1
assert_eq!(estimate_tokens("abcde"), 2); // 5 → ceil(5/4) = 2
assert_eq!(estimate_tokens(&"x".repeat(400)), 100);
}
#[test]
fn context_window_known_and_fallback() {
assert_eq!(context_window_for("qwen3.5:latest"), 262_144);
assert_eq!(context_window_for("kimi-k2:1t"), 1_048_576);
assert_eq!(context_window_for("some-unreleased-model"), DEFAULT_CONTEXT_WINDOW);
}
#[test]
fn budget_passes_well_under_window() {
let check = assert_context_budget(
"qwen3:latest",
&"x".repeat(4_000), // ~1000 tokens
BudgetOpts { max_tokens: Some(500), ..Default::default() },
).expect("well under 40K window");
assert!(check.remaining > 30_000);
}
#[test]
fn budget_fails_when_prompt_overflows_window() {
let huge = "x".repeat(200_000); // ~50K tokens, over qwen3's 40K
let err = assert_context_budget(
"qwen3:latest",
&huge,
BudgetOpts::default(),
).expect_err("should overflow qwen3's 40K window");
assert!(err.1 > 0, "over_by must be positive");
}
#[test]
fn budget_bypass_returns_ok_even_over() {
let huge = "x".repeat(200_000);
let check = assert_context_budget(
"qwen3:latest",
&huge,
BudgetOpts { bypass: true, ..Default::default() },
).expect("bypass must suppress the error");
assert!(check.remaining < 0, "check still reports negative remaining");
}
#[test]
fn budget_counts_system_prompt() {
// 10K-char system prompt → ~2500 tokens. With a big max_tokens
// this should push us closer to the window.
let sys = "s".repeat(10_000);
let prompt = "p".repeat(4_000);
let with_sys = assert_context_budget(
"qwen3:latest",
&prompt,
BudgetOpts {
system: Some(&sys),
max_tokens: Some(500),
..Default::default()
},
).unwrap();
let without_sys = assert_context_budget(
"qwen3:latest",
&prompt,
BudgetOpts { max_tokens: Some(500), ..Default::default() },
).unwrap();
assert!(with_sys.estimated > without_sys.estimated,
"system prompt should raise estimate");
assert_eq!(with_sys.estimated - without_sys.estimated, estimate_tokens(&sys));
}
#[test]
fn overflow_message_includes_numbers() {
let check = BudgetCheck { estimated: 42_000, window: 40_960, remaining: -1_040 };
let msg = overflow_message("qwen3:latest", &check, 3_040, 2_000);
assert!(msg.contains("qwen3:latest"));
assert!(msg.contains("42000t"));
assert!(msg.contains("40960t"));
assert!(msg.contains("3040t"));
}
}
crates/aibridge/src/continuation.rs
+438
View File
@@ -0,0 +1,438 @@
//! Phase 21 — OUTPUT-overflow handler. Ports `generateContinuable`
//! from `tests/multi-agent/agent.ts`.
//!
//! Two failure modes to repair:
//!
//! * EMPTY response — thinking model ate the entire budget on hidden
//! reasoning before emitting a token. Fix: retry the original prompt
//! with 2× the budget, geometric up to `BUDGET_CAP`.
//!
//! * TRUNCATED non-empty — model got most of the way but hit
//! max_tokens before closing the structure. Fix: continue with the
//! partial response in the prompt as scratchpad, so the model knows
//! where to pick up without restarting.
//!
//! `TextGenerator` abstracts the sidecar so tests can inject canned
//! responses without a live Ollama.
use std::future::Future;
use crate::client::{AiClient, GenerateRequest, GenerateResponse};
/// Shape classifier for `is_structurally_complete`. JSON responses
/// must parse; text responses just need to be non-empty.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ResponseShape {
Json,
Text,
}
/// Trait that `generate_continuable` + `generate_tree_split` call. The
/// real implementation forwards to `AiClient::generate`; tests supply a
/// mock with a scripted sequence of responses.
pub trait TextGenerator: Send + Sync {
fn generate_text(
&self,
req: GenerateRequest,
) -> impl Future<Output = Result<GenerateResponse, String>> + Send;
}
impl TextGenerator for AiClient {
fn generate_text(
&self,
req: GenerateRequest,
) -> impl Future<Output = Result<GenerateResponse, String>> + Send {
self.generate(req)
}
}
/// Strip a surrounding ```json``` fence if present. Leaves the inner
/// content alone otherwise. Returns a slice of `s`.
fn strip_json_fence(s: &str) -> &str {
let t = s.trim();
if let Some(rest) = t.strip_prefix("```json") {
rest.trim_start_matches('\n').strip_suffix("```").unwrap_or(rest).trim()
} else if let Some(rest) = t.strip_prefix("```") {
rest.trim_start_matches('\n').strip_suffix("```").unwrap_or(rest).trim()
} else {
t
}
}
/// Port of the TS brace-balance + JSON.parse check. Returns true when
/// the outermost `{...}` block is balanced and parses. Text shape is
/// satisfied by any non-empty, non-whitespace payload.
pub fn is_structurally_complete(text: &str, shape: ResponseShape) -> bool {
if text.trim().is_empty() { return false; }
if shape == ResponseShape::Text { return true; }
let s = strip_json_fence(text);
let Some(start) = s.find('{') else { return false };
let Some(end) = s.rfind('}') else { return false };
if end <= start { return false; }
let slice = &s[start..=end];
// Balance check — cheaper than parse, catches truncated nests.
// String state tracked because `{` inside a string doesn't count.
let mut depth: i32 = 0;
let mut in_str = false;
let mut esc = false;
for c in slice.chars() {
if esc { esc = false; continue; }
if c == '\\' { esc = true; continue; }
if c == '"' { in_str = !in_str; continue; }
if in_str { continue; }
if c == '{' { depth += 1; }
else if c == '}' {
depth -= 1;
if depth < 0 { return false; }
}
}
if depth != 0 { return false; }
// Parse check is the tie-breaker — balanced but invalid JSON (e.g.
// trailing comma before `}`) shouldn't count as complete.
serde_json::from_str::<serde_json::Value>(slice).is_ok()
}
/// Knobs for `generate_continuable`. All optional with sensible
/// defaults that match the TS version.
#[derive(Debug, Clone)]
pub struct ContinuableOpts {
pub model: String,
pub max_tokens: Option<u32>,
pub temperature: Option<f64>,
pub system: Option<String>,
pub shape: ResponseShape,
pub max_continuations: usize,
pub think: Option<bool>,
/// Geometric-backoff ceiling for the empty-response retry path.
/// Matches TS's `budgetCap = 8000`.
pub budget_cap: u32,
/// Maximum empty-response retries before giving up. Matches TS's
/// hardcoded `retry < 3`.
pub max_empty_retries: usize,
}
impl ContinuableOpts {
pub fn new(model: impl Into<String>) -> Self {
Self {
model: model.into(),
max_tokens: None,
temperature: None,
system: None,
shape: ResponseShape::Json,
max_continuations: 3,
think: None,
budget_cap: 8_000,
max_empty_retries: 3,
}
}
}
/// Outcome of a `generate_continuable` call. Carries the combined
/// text plus diagnostic counters so observability downstream can
/// report "how many continuations did that query cost".
#[derive(Debug, Clone)]
pub struct ContinuableOutcome {
pub text: String,
pub empty_retries: usize,
pub continuations: usize,
pub final_complete: bool,
}
fn make_request(opts: &ContinuableOpts, prompt: String, current_max: u32) -> GenerateRequest {
GenerateRequest {
prompt,
model: Some(opts.model.clone()),
system: opts.system.clone(),
temperature: opts.temperature,
max_tokens: Some(current_max),
think: opts.think,
}
}
fn continuation_prompt(original: &str, partial: &str) -> String {
format!(
"{original}\n\n\
PARTIAL RESPONSE SO FAR (continue from here — do NOT restart, \
do NOT repeat what's already there, emit ONLY the remaining \
tokens to close the structure):\n{partial}"
)
}
/// Phase 21 — output-overflow safe generate. See module docs for the
/// two failure modes repaired. On final-still-incomplete, returns the
/// combined text with `final_complete: false` so the caller's parser
/// can throw with the raw text for forensics rather than silently
/// truncating.
pub async fn generate_continuable<G: TextGenerator>(
generator: &G,
prompt: &str,
opts: &ContinuableOpts,
) -> Result<ContinuableOutcome, String> {
let initial_max = opts.max_tokens.unwrap_or(800);
let mut current_max = initial_max;
let mut combined = String::new();
let mut empty_retries = 0usize;
let mut continuations = 0usize;
// Phase 21(a) — empty-response backoff loop.
for retry in 0..opts.max_empty_retries {
let req = make_request(opts, prompt.to_string(), current_max);
let resp = generator.generate_text(req).await?;
if !resp.text.trim().is_empty() {
combined = resp.text;
break;
}
empty_retries = retry + 1;
current_max = (current_max.saturating_mul(2)).min(opts.budget_cap);
}
// Phase 21(b) — structural-completion continuation loop. Runs on
// the truncated-non-empty case; empty + exhausted retries falls
// through with empty combined and final_complete=false.
for _ in 0..opts.max_continuations {
if is_structurally_complete(&combined, opts.shape) {
return Ok(ContinuableOutcome {
text: combined,
empty_retries,
continuations,
final_complete: true,
});
}
if combined.trim().is_empty() {
// Nothing to continue from — continuing "" is identical to
// the initial call and would loop. Bail so the caller sees
// the failure rather than burning N extra calls.
break;
}
let cont_prompt = continuation_prompt(prompt, &combined);
let req = make_request(opts, cont_prompt, current_max.min(opts.budget_cap));
let resp = generator.generate_text(req).await?;
combined.push_str(&resp.text);
continuations += 1;
}
let final_complete = is_structurally_complete(&combined, opts.shape);
Ok(ContinuableOutcome {
text: combined,
empty_retries,
continuations,
final_complete,
})
}
/// Scripted generator for unit tests. Returns responses from `script`
/// in order; extra calls reuse the last entry so tests don't have to
/// count past what they actually assert on.
#[cfg(test)]
pub struct ScriptedGenerator {
script: Vec<Result<String, String>>,
calls: std::sync::Arc<std::sync::Mutex<Vec<GenerateRequest>>>,
}
#[cfg(test)]
impl ScriptedGenerator {
pub fn new<I>(script: I) -> Self
where
I: IntoIterator<Item = Result<String, String>>,
{
Self {
script: script.into_iter().collect(),
calls: std::sync::Arc::new(std::sync::Mutex::new(Vec::new())),
}
}
pub fn calls(&self) -> Vec<GenerateRequest> {
self.calls.lock().unwrap().clone()
}
pub fn call_count(&self) -> usize {
self.calls.lock().unwrap().len()
}
}
#[cfg(test)]
impl TextGenerator for ScriptedGenerator {
fn generate_text(
&self,
req: GenerateRequest,
) -> impl Future<Output = Result<GenerateResponse, String>> + Send {
let i = {
let mut calls = self.calls.lock().unwrap();
let i = calls.len();
calls.push(req.clone());
i
};
let model = req.model.clone().unwrap_or_default();
let entry = self.script.get(i)
.cloned()
.unwrap_or_else(|| self.script.last().cloned().unwrap_or(Ok(String::new())));
async move {
entry.map(|text| GenerateResponse {
text,
model,
tokens_evaluated: None,
tokens_generated: None,
})
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn structural_complete_rejects_empty_and_text_mismatch() {
assert!(!is_structurally_complete("", ResponseShape::Text));
assert!(!is_structurally_complete(" ", ResponseShape::Text));
assert!(is_structurally_complete("any content", ResponseShape::Text));
}
#[test]
fn structural_complete_handles_balanced_json() {
assert!(is_structurally_complete(r#"{"a": 1}"#, ResponseShape::Json));
assert!(is_structurally_complete(
r#"```json
{"a": 1, "b": [1, 2, 3]}
```"#,
ResponseShape::Json,
));
}
#[test]
fn structural_complete_rejects_truncated_json() {
assert!(!is_structurally_complete(r#"{"a": 1"#, ResponseShape::Json));
assert!(!is_structurally_complete(r#"{"a": {"b": 1"#, ResponseShape::Json));
// Trailing comma — balanced but unparseable
assert!(!is_structurally_complete(r#"{"a": 1,}"#, ResponseShape::Json));
}
#[test]
fn structural_complete_ignores_braces_inside_strings() {
assert!(is_structurally_complete(r#"{"s": "has } inside"}"#, ResponseShape::Json));
assert!(is_structurally_complete(r#"{"s": "escaped \" quote"}"#, ResponseShape::Json));
}
#[tokio::test]
async fn continuable_returns_first_response_when_complete() {
let generator = ScriptedGenerator::new(vec![Ok(r#"{"ok": true}"#.to_string())]);
let opts = ContinuableOpts::new("qwen3:latest");
let out = generate_continuable(&generator, "test", &opts).await.unwrap();
assert!(out.final_complete);
assert_eq!(out.empty_retries, 0);
assert_eq!(out.continuations, 0);
assert_eq!(generator.call_count(), 1);
}
#[tokio::test]
async fn continuable_retries_on_empty_with_doubled_budget() {
// Two empties, then a good response. Third call should see 4×
// the initial budget (2× twice).
let generator = ScriptedGenerator::new(vec![
Ok("".to_string()),
Ok("".to_string()),
Ok(r#"{"ok": true}"#.to_string()),
]);
let mut opts = ContinuableOpts::new("qwen3:latest");
opts.max_tokens = Some(100);
let out = generate_continuable(&generator, "test", &opts).await.unwrap();
assert!(out.final_complete);
assert_eq!(out.empty_retries, 2);
let calls = generator.calls();
assert_eq!(calls.len(), 3);
assert_eq!(calls[0].max_tokens, Some(100));
assert_eq!(calls[1].max_tokens, Some(200));
assert_eq!(calls[2].max_tokens, Some(400));
}
#[tokio::test]
async fn continuable_caps_budget_at_budget_cap() {
let generator = ScriptedGenerator::new(vec![
Ok("".to_string()),
Ok("".to_string()),
Ok(r#"{"ok":1}"#.to_string()),
]);
let mut opts = ContinuableOpts::new("qwen3:latest");
opts.max_tokens = Some(5_000);
opts.budget_cap = 7_000;
let out = generate_continuable(&generator, "test", &opts).await.unwrap();
assert!(out.final_complete);
let calls = generator.calls();
// 5000 → doubled would be 10000; cap pulls it to 7000.
assert_eq!(calls[1].max_tokens, Some(7_000));
assert_eq!(calls[2].max_tokens, Some(7_000));
}
#[tokio::test]
async fn continuable_glues_truncated_response() {
// First call returns balanced-open `{...`; continuation closes
// it with `...}`. Combined must parse.
let generator = ScriptedGenerator::new(vec![
Ok(r#"{"fills": [{"candidate_id": "C-001""#.to_string()),
Ok(r#", "name": "Alice"}]}"#.to_string()),
]);
let opts = ContinuableOpts::new("qwen3:latest");
let out = generate_continuable(&generator, "ORIGINAL", &opts).await.unwrap();
assert!(out.final_complete, "combined must parse: {}", out.text);
assert_eq!(out.continuations, 1);
let calls = generator.calls();
assert_eq!(calls.len(), 2);
// Continuation prompt must contain the partial — that's the
// scratchpad primitive J called out.
let cont_prompt = &calls[1].prompt;
assert!(cont_prompt.contains("ORIGINAL"),
"continuation must include original prompt");
assert!(cont_prompt.contains(r#"{"fills": [{"candidate_id": "C-001""#),
"continuation must include partial");
}
#[tokio::test]
async fn continuable_does_not_loop_on_persistent_empty() {
// All three retries return empty; we must NOT then enter the
// continuation loop with an empty partial (would burn 3 more
// calls continuing from "").
let generator = ScriptedGenerator::new(vec![
Ok("".to_string()),
Ok("".to_string()),
Ok("".to_string()),
]);
let opts = ContinuableOpts::new("qwen3:latest");
let out = generate_continuable(&generator, "test", &opts).await.unwrap();
assert!(!out.final_complete);
assert_eq!(out.empty_retries, 3);
assert_eq!(out.continuations, 0, "must not continue from empty");
assert_eq!(generator.call_count(), 3);
}
#[tokio::test]
async fn continuable_returns_raw_on_exhausted_continuations() {
// Three continuations that never complete — caller's parser
// will throw. We must return the combined text so forensics
// has the raw content, not a lossy truncation.
let generator = ScriptedGenerator::new(vec![
Ok(r#"{"a": ["#.to_string()),
Ok("1,".to_string()),
Ok("2,".to_string()),
Ok("3,".to_string()),
]);
let mut opts = ContinuableOpts::new("qwen3:latest");
opts.max_continuations = 3;
let out = generate_continuable(&generator, "test", &opts).await.unwrap();
assert!(!out.final_complete);
assert_eq!(out.continuations, 3);
assert!(out.text.contains(r#"{"a": ["#));
assert!(out.text.contains("3,"));
}
#[tokio::test]
async fn continuable_propagates_generator_errors() {
let generator = ScriptedGenerator::new(vec![Err("sidecar 503".to_string())]);
let opts = ContinuableOpts::new("qwen3:latest");
let err = generate_continuable(&generator, "test", &opts).await.unwrap_err();
assert!(err.contains("503"));
}
}
crates/aibridge/src/lib.rs
+3
View File
@@ -1,2 +1,5 @@
pub mod client; pub mod client;
pub mod context;
pub mod continuation;
pub mod service; pub mod service;
pub mod tree_split;
crates/aibridge/src/tree_split.rs
+326
View File
@@ -0,0 +1,326 @@
//! Phase 21 — INPUT-overflow handler. Ports `generateTreeSplit` from
//! `tests/multi-agent/agent.ts`.
//!
//! When the input corpus exceeds the model's window (200 playbooks
//! pasted into a T4 strategic prompt, a long retrospective digest, a
//! cross-corpus summarization), raising `max_tokens` doesn't help —
//! the prompt itself is the problem. The answer is map-reduce:
//!
//! 1. Caller shards the input at semantic boundaries (records,
//! paragraphs, playbook entries).
//! 2. For each shard, build a map prompt that includes the running
//! scratchpad and run it through `generate_continuable`.
//! 3. Append the map output to the scratchpad (oldest-first
//! truncation when it outgrows `scratchpad_budget`).
//! 4. Build a reduce prompt from the final scratchpad and run it.
//!
//! Every shard prompt and the reduce prompt go through
//! `assert_context_budget` first — if a single shard still overflows
//! we bubble the error up rather than silently truncating. That's the
//! whole point of Phase 21.
use crate::context::{assert_context_budget, BudgetOpts, estimate_tokens, overflow_message,
DEFAULT_MAX_TOKENS, DEFAULT_SAFETY_MARGIN};
use crate::continuation::{generate_continuable, ContinuableOpts, ResponseShape, TextGenerator};
/// Callback signatures — caller supplies closures that stitch the
/// scratchpad into each shard's prompt and build the final reduce
/// prompt. Kept as `Fn` (not `FnMut`) so the map loop can call them
/// by reference.
pub type MapPromptFn<'a> = dyn Fn(&str, &str) -> String + Send + Sync + 'a;
pub type ReducePromptFn<'a> = dyn Fn(&str) -> String + Send + Sync + 'a;
/// Knobs for `generate_tree_split`.
#[derive(Debug, Clone)]
pub struct TreeSplitOpts {
pub model: String,
pub system: Option<String>,
pub temperature: Option<f64>,
/// max_tokens for map AND reduce (reduce defaults are usually
/// higher; caller overrides for just reduce by calling through
/// continuable directly if needed).
pub max_tokens: Option<u32>,
pub reduce_max_tokens: Option<u32>,
pub think: Option<bool>,
/// Soft ceiling on scratchpad size (estimated tokens). When it
/// grows past this, the oldest shard digest gets dropped. Default
/// 6000, matching the TS implementation.
pub scratchpad_budget: usize,
pub safety_margin: Option<usize>,
}
impl TreeSplitOpts {
pub fn new(model: impl Into<String>) -> Self {
Self {
model: model.into(),
system: None,
temperature: None,
max_tokens: None,
reduce_max_tokens: None,
think: None,
scratchpad_budget: 6_000,
safety_margin: None,
}
}
}
/// Result — final reduce response plus the accumulated scratchpad so
/// the caller can inspect what was kept vs truncated.
#[derive(Debug, Clone)]
pub struct TreeSplitResult {
pub response: String,
pub scratchpad: String,
pub shards_processed: usize,
pub scratchpad_truncations: usize,
pub total_continuations: usize,
}
/// Drop shard-digest blocks from the head of `scratchpad` until its
/// estimated-token count fits the budget. Digest blocks are delimited
/// by `\n— shard N/M digest —\n` so we can find the first one and
/// chop everything before its successor.
fn truncate_scratchpad(scratchpad: &mut String, budget_tokens: usize) -> bool {
if estimate_tokens(scratchpad) <= budget_tokens { return false; }
// Find the second delimiter — everything before it gets dropped.
const DELIM_PREFIX: &str = "\n— shard ";
let mut cursor = 0;
let mut truncated = false;
while estimate_tokens(&scratchpad[cursor..]) > budget_tokens {
// Skip past a leading delimiter (if we're sitting on one from
// a previous iteration), then find the next.
let search_from = cursor + if scratchpad[cursor..].starts_with(DELIM_PREFIX) {
DELIM_PREFIX.len()
} else { 0 };
let Some(rel_next) = scratchpad[search_from..].find(DELIM_PREFIX) else { break };
cursor = search_from + rel_next;
truncated = true;
}
if cursor > 0 {
scratchpad.drain(..cursor);
}
truncated
}
/// Phase 21 — map-reduce over shards with a running scratchpad. See
/// module docs.
pub async fn generate_tree_split<G: TextGenerator>(
generator: &G,
shards: &[String],
map_prompt: &MapPromptFn<'_>,
reduce_prompt: &ReducePromptFn<'_>,
opts: &TreeSplitOpts,
) -> Result<TreeSplitResult, String> {
let mut scratchpad = String::new();
let safety = opts.safety_margin.unwrap_or(DEFAULT_SAFETY_MARGIN);
let map_max = opts.max_tokens.unwrap_or(DEFAULT_MAX_TOKENS as u32);
let reduce_max = opts.reduce_max_tokens.unwrap_or(1_500);
let mut truncations = 0usize;
let mut total_continuations = 0usize;
for (i, shard) in shards.iter().enumerate() {
let shard_prompt = map_prompt(shard, &scratchpad);
// Loud-fail on per-shard overflow — caller sharded too
// coarsely. Silent truncation is exactly the mode J rejected.
let budget = BudgetOpts {
system: opts.system.as_deref(),
max_tokens: Some(map_max as usize),
safety_margin: Some(safety),
bypass: false,
};
let check = assert_context_budget(&opts.model, &shard_prompt, budget)
.map_err(|(c, over)| overflow_message(&opts.model, &c, over, safety))?;
let _ = check;
let mut cont_opts = ContinuableOpts::new(&opts.model);
cont_opts.max_tokens = Some(map_max);
cont_opts.temperature = opts.temperature;
cont_opts.system = opts.system.clone();
cont_opts.shape = ResponseShape::Text;
cont_opts.think = opts.think;
let outcome = generate_continuable(generator, &shard_prompt, &cont_opts).await?;
total_continuations += outcome.continuations;
// Append this shard's digest and, if needed, drop oldest.
scratchpad.push_str(&format!(
"\n— shard {}/{} digest —\n{}",
i + 1, shards.len(), outcome.text.trim(),
));
if truncate_scratchpad(&mut scratchpad, opts.scratchpad_budget) {
truncations += 1;
}
}
// Reduce pass. Budget check first — if the scratchpad is still too
// big for the reduce prompt we fail loud with numbers.
let reduce_p = reduce_prompt(&scratchpad);
let budget = BudgetOpts {
system: opts.system.as_deref(),
max_tokens: Some(reduce_max as usize),
safety_margin: Some(safety),
bypass: false,
};
assert_context_budget(&opts.model, &reduce_p, budget)
.map_err(|(c, over)| overflow_message(&opts.model, &c, over, safety))?;
let mut cont_opts = ContinuableOpts::new(&opts.model);
cont_opts.max_tokens = Some(reduce_max);
cont_opts.temperature = opts.temperature;
cont_opts.system = opts.system.clone();
cont_opts.shape = ResponseShape::Text;
cont_opts.think = opts.think;
let outcome = generate_continuable(generator, &reduce_p, &cont_opts).await?;
total_continuations += outcome.continuations;
Ok(TreeSplitResult {
response: outcome.text,
scratchpad,
shards_processed: shards.len(),
scratchpad_truncations: truncations,
total_continuations,
})
}
#[cfg(test)]
mod tests {
use super::*;
use crate::continuation::ScriptedGenerator;
fn simple_map(shard: &str, scratchpad: &str) -> String {
format!("SCRATCHPAD:\n{scratchpad}\n---\nSHARD:\n{shard}\n---\nDIGEST:")
}
fn simple_reduce(scratchpad: &str) -> String {
format!("SCRATCHPAD:\n{scratchpad}\n---\nFINAL:")
}
#[tokio::test]
async fn tree_split_runs_map_then_reduce() {
// 3 shards → 3 map calls → 1 reduce call = 4 responses scripted.
let generator = ScriptedGenerator::new(vec![
Ok("digest-1".to_string()),
Ok("digest-2".to_string()),
Ok("digest-3".to_string()),
Ok("FINAL ANSWER".to_string()),
]);
let shards: Vec<String> = ["a", "b", "c"].iter().map(|s| s.to_string()).collect();
let opts = TreeSplitOpts::new("qwen3:latest");
let map_fn: &MapPromptFn = &simple_map;
let reduce_fn: &ReducePromptFn = &simple_reduce;
let result = generate_tree_split(&generator, &shards, map_fn, reduce_fn, &opts)
.await
.unwrap();
assert_eq!(result.shards_processed, 3);
assert_eq!(result.response, "FINAL ANSWER");
assert_eq!(generator.call_count(), 4);
// Scratchpad must carry all three digests in order.
assert!(result.scratchpad.contains("digest-1"));
assert!(result.scratchpad.contains("digest-2"));
assert!(result.scratchpad.contains("digest-3"));
}
#[tokio::test]
async fn tree_split_reduce_prompt_sees_full_scratchpad() {
let generator = ScriptedGenerator::new(vec![
Ok("summary-A".to_string()),
Ok("summary-B".to_string()),
Ok("REDUCED".to_string()),
]);
let shards = vec!["input-one".to_string(), "input-two".to_string()];
let opts = TreeSplitOpts::new("qwen3:latest");
let _ = generate_tree_split(&generator, &shards, &simple_map, &simple_reduce, &opts)
.await
.unwrap();
// Third call = reduce. Its prompt must include both digests.
let calls = generator.calls();
let reduce_prompt = &calls[2].prompt;
assert!(reduce_prompt.contains("summary-A"),
"reduce prompt must see first shard digest");
assert!(reduce_prompt.contains("summary-B"),
"reduce prompt must see second shard digest");
}
#[tokio::test]
async fn tree_split_loud_fails_on_shard_overflow() {
let generator = ScriptedGenerator::new(vec![Ok("digest".to_string())]);
// One gigantic shard — well over qwen3's 40K window even as a
// prompt. The budget check must reject before any generate call.
let shards = vec!["x".repeat(200_000)];
let opts = TreeSplitOpts::new("qwen3:latest");
let err = generate_tree_split(&generator, &shards, &simple_map, &simple_reduce, &opts)
.await
.expect_err("shard-sized overflow must be rejected");
assert!(err.contains("overflow"), "error should mention overflow: {err}");
assert_eq!(generator.call_count(), 0, "generate must not be called on overflow");
}
#[tokio::test]
async fn tree_split_truncates_scratchpad_when_over_budget() {
// Tight budget so each shard trips truncation. qwen3's 40K
// window is fine; the budget we care about is the scratchpad
// cap, not the model window.
let generator = ScriptedGenerator::new(vec![
Ok("A".repeat(2_000)),
Ok("B".repeat(2_000)),
Ok("C".repeat(2_000)),
Ok("D".repeat(2_000)),
Ok("FINAL".to_string()),
]);
let shards: Vec<String> = (0..4).map(|i| format!("shard{i}")).collect();
let mut opts = TreeSplitOpts::new("qwen3:latest");
opts.scratchpad_budget = 1_000; // ~4000 chars — one digest barely fits
let result = generate_tree_split(&generator, &shards, &simple_map, &simple_reduce, &opts)
.await
.unwrap();
assert!(result.scratchpad_truncations > 0,
"tight budget must trigger truncation");
// Scratchpad should still fit roughly within the budget
// (post-truncation); the estimator uses chars/4 so the bound
// is ~budget*4 chars. Give some slack for the delimiter.
let scratchpad_tokens = estimate_tokens(&result.scratchpad);
assert!(scratchpad_tokens <= opts.scratchpad_budget * 2,
"scratchpad {} tokens vs budget {}", scratchpad_tokens, opts.scratchpad_budget);
}
#[tokio::test]
async fn tree_split_reports_continuations_from_map_and_reduce() {
// First shard: truncated-then-continued. Reduce: truncated-then-continued.
// 1 shard: 2 map calls (initial + continuation), then 2 reduce calls.
let generator = ScriptedGenerator::new(vec![
Ok("partial".to_string()), // map shape=text, non-empty → complete on first pass
Ok("reduce-out".to_string()),
]);
let shards = vec!["only".to_string()];
let opts = TreeSplitOpts::new("qwen3:latest");
let result = generate_tree_split(&generator, &shards, &simple_map, &simple_reduce, &opts)
.await
.unwrap();
// Text shape treats non-empty as complete → 0 continuations.
assert_eq!(result.total_continuations, 0);
assert_eq!(result.shards_processed, 1);
}
#[test]
fn truncate_scratchpad_noop_when_under_budget() {
let mut s = "\n— shard 1/1 digest —\nshort".to_string();
let truncated = truncate_scratchpad(&mut s, 1_000);
assert!(!truncated);
assert!(s.contains("short"));
}
#[test]
fn truncate_scratchpad_drops_oldest_first() {
let mut s = format!(
"\n— shard 1/3 digest —\n{}\n— shard 2/3 digest —\n{}\n— shard 3/3 digest —\nshort",
"x".repeat(4_000), // ~1000 tokens
"y".repeat(4_000), // ~1000 tokens
);
let truncated = truncate_scratchpad(&mut s, 500); // ~2000 chars
assert!(truncated);
assert!(!s.contains(&"x".repeat(4_000)),
"oldest digest should be dropped");
assert!(s.contains("short"),
"newest digest should survive");
}
}
crates/catalogd/src/grpc.rs
+10 -1
View File
@@ -54,7 +54,16 @@ impl CatalogService for CatalogGrpc {
let manifest = self.registry let manifest = self.registry
.register(req.name, SchemaFingerprint(req.schema_fingerprint), objects) .register(req.name, SchemaFingerprint(req.schema_fingerprint), objects)
.await .await
.map_err(|e| Status::internal(e))?; .map_err(|e| {
// Mirror the HTTP side's 409 mapping: schema drift is a
// precondition failure, not a server error. Keeps gRPC
// and HTTP callers seeing the same diagnostic signal.
if e.contains("different schema") {
Status::failed_precondition(e)
} else {
Status::internal(e)
}
})?;
Ok(Response::new(manifest_to_proto(&manifest))) Ok(Response::new(manifest_to_proto(&manifest)))
} }
crates/catalogd/src/registry.rs
+452 -3
View File
@@ -28,6 +28,27 @@ pub struct MigrateBucketsReport {
pub manifests_persisted: usize, pub manifests_persisted: usize,
} }
#[derive(Debug, Clone, Default, serde::Serialize)]
pub struct DedupeReport {
pub groups: usize,
pub removed: usize,
pub kept: Vec<DedupeKept>,
pub errors: Vec<DedupeError>,
}
#[derive(Debug, Clone, serde::Serialize)]
pub struct DedupeKept {
pub name: String,
pub kept_id: String,
pub removed: usize,
}
#[derive(Debug, Clone, serde::Serialize)]
pub struct DedupeError {
pub manifest_id: String,
pub error: String,
}
/// Partial metadata update — only set fields are applied. /// Partial metadata update — only set fields are applied.
#[derive(Debug, Clone, Default, serde::Deserialize)] #[derive(Debug, Clone, Default, serde::Deserialize)]
pub struct MetadataUpdate { pub struct MetadataUpdate {
@@ -212,13 +233,65 @@ impl Registry {
Ok(()) Ok(())
} }
/// Register a new dataset. Persists manifest to storage before updating memory. /// Register a dataset. Idempotent on `name`:
/// - No existing dataset by this name → create new manifest.
/// - Exists with same schema fingerprint → update objects + `updated_at`
/// in place (re-ingest of identical-shape data).
/// - Exists with different schema fingerprint → reject as `Err`
/// (callers map to HTTP 409 / gRPC FAILED_PRECONDITION). Schema
/// evolution is not yet exposed as an HTTP endpoint; callers must
/// register under a new name or delete the existing manifest first.
///
/// Concurrency: the write lock is held across the storage write. That
/// serializes concurrent registers but is correctness-mandatory —
/// dropping the lock across the check→insert sequence creates a TOCTOU
/// window where two callers with the same name both see "no existing"
/// and both insert, reintroducing the duplicate-manifest bug this
/// function exists to prevent. Register is a metadata-only hop on the
/// ingest path, so the serialization cost is acceptable. Precedent:
/// `update_metadata` also holds the write lock across its I/O.
///
/// Legacy state: if multiple manifests with the same name already exist
/// (from before this was idempotent), the most recently updated one is
/// treated as canonical. Run `dedupe_by_name` to clean up the rest.
pub async fn register( pub async fn register(
&self, &self,
name: String, name: String,
schema_fingerprint: SchemaFingerprint, schema_fingerprint: SchemaFingerprint,
objects: Vec<ObjectRef>, objects: Vec<ObjectRef>,
) -> Result<DatasetManifest, String> { ) -> Result<DatasetManifest, String> {
let mut datasets = self.datasets.write().await;
let existing = datasets
.values()
.filter(|d| d.name == name)
.max_by_key(|d| d.updated_at)
.cloned();
if let Some(mut manifest) = existing {
if manifest.schema_fingerprint != schema_fingerprint {
return Err(format!(
"dataset '{}' already exists with a different schema \
(existing fingerprint: {}, new: {}). Schema-evolution \
migration is not yet exposed as an HTTP endpoint; \
register under a new name or delete the existing \
manifest first.",
name, manifest.schema_fingerprint.0, schema_fingerprint.0,
));
}
manifest.objects = objects;
manifest.updated_at = chrono::Utc::now();
let manifest_key = format!("{MANIFEST_PREFIX}/{}.json", manifest.id);
let json = serde_json::to_vec_pretty(&manifest).map_err(|e| e.to_string())?;
ops::put(&self.store, &manifest_key, json.into()).await?;
datasets.insert(manifest.id.clone(), manifest.clone());
tracing::info!("re-registered (idempotent): {} ({})", manifest.name, manifest.id);
return Ok(manifest);
}
let now = chrono::Utc::now(); let now = chrono::Utc::now();
let manifest = DatasetManifest { let manifest = DatasetManifest {
id: DatasetId::new(), id: DatasetId::new(),
@@ -240,12 +313,10 @@ impl Registry {
embedding_refresh_policy: None, embedding_refresh_policy: None,
}; };
// Write-ahead: persist before in-memory update
let manifest_key = format!("{MANIFEST_PREFIX}/{}.json", manifest.id); let manifest_key = format!("{MANIFEST_PREFIX}/{}.json", manifest.id);
let json = serde_json::to_vec_pretty(&manifest).map_err(|e| e.to_string())?; let json = serde_json::to_vec_pretty(&manifest).map_err(|e| e.to_string())?;
ops::put(&self.store, &manifest_key, json.into()).await?; ops::put(&self.store, &manifest_key, json.into()).await?;
let mut datasets = self.datasets.write().await;
datasets.insert(manifest.id.clone(), manifest.clone()); datasets.insert(manifest.id.clone(), manifest.clone());
tracing::info!("registered dataset: {} ({})", manifest.name, manifest.id); tracing::info!("registered dataset: {} ({})", manifest.name, manifest.id);
@@ -380,6 +451,72 @@ impl Registry {
Ok(manifest.clone()) Ok(manifest.clone())
} }
/// Collapse duplicate manifests that share a `name`. Winner per group:
/// 1. Prefer a manifest with a non-null `row_count` (already resynced).
/// 2. Break ties by newest `updated_at`.
/// Losers are removed from the in-memory registry AND their manifest
/// JSON is deleted from object storage. Datasets with a single manifest
/// are untouched. Parquet data files are never touched — only catalog
/// metadata. See `register` for the prevention side of this fix.
///
/// Operator-only. Do not run while ingest is active: a concurrent
/// `register` between the snapshot and the delete sweep can create a
/// new manifest under a name we just deduped, producing a transient
/// count of 2 until the next dedupe run. Safe outside ingest windows.
pub async fn dedupe_by_name(&self) -> DedupeReport {
let groups: Vec<(String, Vec<DatasetManifest>)> = {
let datasets = self.datasets.read().await;
let mut by_name: HashMap<String, Vec<DatasetManifest>> = HashMap::new();
for m in datasets.values() {
by_name.entry(m.name.clone()).or_default().push(m.clone());
}
by_name.into_iter().filter(|(_, v)| v.len() > 1).collect()
};
let mut report = DedupeReport::default();
let mut to_delete: Vec<DatasetId> = Vec::new();
for (name, mut manifests) in groups {
report.groups += 1;
manifests.sort_by(|a, b| {
b.row_count.is_some().cmp(&a.row_count.is_some())
.then(b.updated_at.cmp(&a.updated_at))
});
let winner = &manifests[0];
report.kept.push(DedupeKept {
name: name.clone(),
kept_id: winner.id.to_string(),
removed: manifests.len() - 1,
});
for loser in &manifests[1..] {
to_delete.push(loser.id.clone());
}
}
{
let mut datasets = self.datasets.write().await;
for id in &to_delete {
datasets.remove(id);
}
}
for id in &to_delete {
let key = format!("{MANIFEST_PREFIX}/{}.json", id);
match ops::delete(&self.store, &key).await {
Ok(_) => report.removed += 1,
Err(e) => report.errors.push(DedupeError {
manifest_id: id.to_string(),
error: e,
}),
}
}
tracing::info!(
"dedupe: {} groups collapsed, {} manifests removed, {} errors",
report.groups, report.removed, report.errors.len(),
);
report
}
/// Resync every dataset that currently has a null row_count. /// Resync every dataset that currently has a null row_count.
/// Returns (successes, failures) where each entry is (name, detail). /// Returns (successes, failures) where each entry is (name, detail).
pub async fn resync_missing(&self) -> (Vec<(String, u64)>, Vec<(String, String)>) { pub async fn resync_missing(&self) -> (Vec<(String, u64)>, Vec<(String, String)>) {
@@ -549,6 +686,46 @@ impl Registry {
Ok(()) Ok(())
} }
/// Remove a dataset from the catalog by name. Deletes the manifest
/// from both the in-memory registry and object storage.
///
/// Scope is metadata-only: the underlying parquet files, vector
/// indexes, tombstones, trial journals, and AiViews that reference
/// this dataset are NOT touched. Caller is responsible for any
/// cascade cleanup. This mirrors how a DROP TABLE in a typical
/// warehouse separates "forget about this dataset" from "actually
/// reclaim the bytes".
///
/// Returns Ok(()) when the dataset existed and was removed; an Err
/// with "dataset not found" when no manifest by that name exists.
/// Legacy state with duplicate names: all matching manifests are
/// removed (effectively a post-hoc dedupe on that name).
pub async fn delete_dataset(&self, name: &str) -> Result<usize, String> {
let ids_to_remove: Vec<DatasetId> = {
let datasets = self.datasets.read().await;
datasets.values().filter(|d| d.name == name).map(|d| d.id.clone()).collect()
};
if ids_to_remove.is_empty() {
return Err(format!("dataset not found: {name}"));
}
for id in &ids_to_remove {
let key = format!("{MANIFEST_PREFIX}/{}.json", id);
if let Err(e) = ops::delete(&self.store, &key).await {
// Storage delete failed — log and keep going. The in-memory
// remove below is still correct; on restart, the missing
// storage object just means nothing to rehydrate.
tracing::warn!("delete_dataset '{name}': storage delete of {key} failed: {e}");
}
}
let mut datasets = self.datasets.write().await;
for id in &ids_to_remove {
datasets.remove(id);
}
tracing::info!("deleted dataset '{}' ({} manifest(s))", name, ids_to_remove.len());
Ok(ids_to_remove.len())
}
/// List datasets whose `embedding_stale_since` is set — they need a refresh. /// List datasets whose `embedding_stale_since` is set — they need a refresh.
pub async fn stale_datasets(&self) -> Vec<DatasetManifest> { pub async fn stale_datasets(&self) -> Vec<DatasetManifest> {
let datasets = self.datasets.read().await; let datasets = self.datasets.read().await;
@@ -579,3 +756,275 @@ impl Registry {
Ok(manifest.clone()) Ok(manifest.clone())
} }
} }
#[cfg(test)]
mod tests {
use super::*;
use object_store::memory::InMemory;
fn fixture() -> Registry {
Registry::new(Arc::new(InMemory::new()))
}
fn fp(s: &str) -> SchemaFingerprint {
SchemaFingerprint(s.to_string())
}
fn obj(key: &str) -> ObjectRef {
ObjectRef {
bucket: "primary".to_string(),
key: key.to_string(),
size_bytes: 100,
created_at: chrono::Utc::now(),
}
}
#[tokio::test]
async fn fresh_register_creates_new_manifest() {
let reg = fixture();
let m = reg.register("a".into(), fp("f1"), vec![obj("a.parquet")]).await.unwrap();
assert_eq!(m.name, "a");
assert_eq!(m.schema_fingerprint, fp("f1"));
assert_eq!(reg.list().await.len(), 1);
assert_eq!(reg.get_by_name("a").await.unwrap().id, m.id);
}
#[tokio::test]
async fn re_register_same_fingerprint_is_idempotent() {
let reg = fixture();
let first = reg.register("a".into(), fp("f1"), vec![obj("old.parquet")]).await.unwrap();
let second = reg.register("a".into(), fp("f1"), vec![obj("new.parquet")]).await.unwrap();
assert_eq!(first.id, second.id, "same name+fp should reuse ID");
assert_eq!(reg.list().await.len(), 1, "no duplicate manifest created");
let fetched = reg.get_by_name("a").await.unwrap();
assert_eq!(fetched.objects[0].key, "new.parquet", "objects replaced");
assert!(fetched.updated_at >= first.updated_at, "updated_at bumped");
}
#[tokio::test]
async fn re_register_different_fingerprint_rejects() {
let reg = fixture();
reg.register("a".into(), fp("f1"), vec![obj("a.parquet")]).await.unwrap();
let err = reg.register("a".into(), fp("f2"), vec![obj("a.parquet")]).await.unwrap_err();
assert!(err.contains("different schema"), "error mentions schema drift: {err}");
assert_eq!(reg.list().await.len(), 1, "failed register did not mutate state");
assert_eq!(reg.get_by_name("a").await.unwrap().schema_fingerprint, fp("f1"));
}
#[tokio::test]
async fn delete_dataset_removes_manifest_and_in_memory_entry() {
let reg = fixture();
reg.register("to_delete".into(), fp("f1"), vec![obj("d.parquet")]).await.unwrap();
reg.register("keepme".into(), fp("f2"), vec![obj("k.parquet")]).await.unwrap();
assert_eq!(reg.list().await.len(), 2);
let removed = reg.delete_dataset("to_delete").await.unwrap();
assert_eq!(removed, 1, "one manifest removed");
assert!(reg.get_by_name("to_delete").await.is_none());
assert!(reg.get_by_name("keepme").await.is_some(), "unrelated dataset untouched");
assert_eq!(reg.list().await.len(), 1);
}
#[tokio::test]
async fn delete_dataset_404s_on_unknown_name() {
let reg = fixture();
let err = reg.delete_dataset("nope").await.unwrap_err();
assert!(err.starts_with("dataset not found"), "error wording: {err}");
}
#[tokio::test]
async fn delete_dataset_removes_all_dupes_with_matching_name() {
// Legacy zombies: direct insert so we can seed duplicates,
// mirroring pre-idempotent-register state. delete_dataset should
// sweep every manifest sharing that name in a single call.
let reg = fixture();
let now = chrono::Utc::now();
for _ in 0..3 {
let m = DatasetManifest {
id: DatasetId::new(),
name: "zombie".into(),
schema_fingerprint: fp("fp"),
objects: vec![obj("z.parquet")],
created_at: now, updated_at: now,
row_count: None, description: String::new(), owner: String::new(),
sensitivity: None, columns: vec![], lineage: None, freshness: None,
tags: vec![], last_embedded_at: None, embedding_stale_since: None,
embedding_refresh_policy: None,
};
reg.datasets.write().await.insert(m.id.clone(), m);
}
assert_eq!(reg.list().await.len(), 3);
let removed = reg.delete_dataset("zombie").await.unwrap();
assert_eq!(removed, 3, "all three manifests removed in one sweep");
assert_eq!(reg.list().await.len(), 0);
}
#[tokio::test]
async fn dedupe_no_duplicates_is_noop() {
let reg = fixture();
reg.register("a".into(), fp("f1"), vec![obj("a.parquet")]).await.unwrap();
reg.register("b".into(), fp("f2"), vec![obj("b.parquet")]).await.unwrap();
let report = reg.dedupe_by_name().await;
assert_eq!(report.groups, 0);
assert_eq!(report.removed, 0);
assert_eq!(reg.list().await.len(), 2);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 8)]
async fn concurrent_register_stress_many_workers_one_manifest() {
// Stress variant: fire 32 concurrent registers for the same new
// name across 8 worker threads. Single-manifest invariant must
// hold regardless of how the scheduler interleaves them. This
// catches race regressions the 2-call variant below might miss
// under a forgiving scheduler.
let reg = fixture();
let mut handles = Vec::new();
for i in 0..32 {
let r = reg.clone();
handles.push(tokio::spawn(async move {
r.register(
"stress".into(),
fp("stress-fp"),
vec![obj(&format!("{i}.parquet"))],
).await
}));
}
for h in handles {
h.await.unwrap().expect("each register succeeds (idempotent)");
}
let all = reg.list().await;
let stress_manifests: Vec<_> = all.iter().filter(|d| d.name == "stress").collect();
assert_eq!(stress_manifests.len(), 1,
"32 concurrent registers produced {} manifests for 'stress' — expected 1",
stress_manifests.len());
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn concurrent_register_same_new_name_collapses_to_one_manifest() {
// Pins the TOCTOU fix: without the write-lock held across the
// check→insert sequence, two parallel registers of the same
// previously-unknown name would both see "no existing" and both
// insert. Multi-thread flavor is required to actually exercise
// the race — under single-threaded tokio the scheduler doesn't
// preempt between awaits on the same task's critical section.
// Under the current impl, whichever acquires the write lock
// first creates the manifest; the second observes it and takes
// the idempotent-update branch.
let reg = fixture();
let a = reg.clone();
let b = reg.clone();
let (ra, rb) = tokio::join!(
async move {
a.register("race".into(), fp("shared-fp"), vec![obj("a.parquet")]).await
},
async move {
b.register("race".into(), fp("shared-fp"), vec![obj("b.parquet")]).await
},
);
let m_a = ra.expect("first register succeeds");
let m_b = rb.expect("second register succeeds (same fp = idempotent)");
assert_eq!(m_a.id, m_b.id, "both calls resolve to a single DatasetId");
assert_eq!(reg.list().await.len(), 1, "no duplicate manifest created");
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn concurrent_register_same_name_different_fp_one_wins_one_rejects() {
// Reverse case: concurrent registers with conflicting fingerprints.
// Deterministic outcome: one call's fingerprint gets persisted as
// canonical, the other is rejected on seeing the now-established
// fingerprint. Which one wins is scheduler-dependent — we only
// assert exactly one success and exactly one rejection.
let reg = fixture();
let a = reg.clone();
let b = reg.clone();
let (ra, rb) = tokio::join!(
async move { a.register("race".into(), fp("fp1"), vec![obj("a.parquet")]).await },
async move { b.register("race".into(), fp("fp2"), vec![obj("b.parquet")]).await },
);
let successes = [&ra, &rb].iter().filter(|r| r.is_ok()).count();
let rejections = [&ra, &rb].iter().filter(|r| r.is_err()).count();
assert_eq!(successes, 1, "exactly one register wins");
assert_eq!(rejections, 1, "the other is rejected on fingerprint");
let rejection_msg = [&ra, &rb].iter()
.find_map(|r| r.as_ref().err())
.unwrap();
assert!(rejection_msg.contains("different schema"));
assert_eq!(reg.list().await.len(), 1);
}
#[tokio::test]
async fn dedupe_collapses_dupes_preferring_non_null_row_count() {
let reg = fixture();
// Simulate legacy pre-idempotent-register state by inserting
// manifests with the same `name` directly. Mirrors what happened
// with the 308× successful_playbooks duplication.
let now = chrono::Utc::now();
let old_with_rows = DatasetManifest {
id: DatasetId::new(),
name: "dupes".into(),
schema_fingerprint: fp("f1"),
objects: vec![obj("dupes.parquet")],
created_at: now - chrono::Duration::hours(2),
updated_at: now - chrono::Duration::hours(2),
row_count: Some(42),
description: String::new(),
owner: String::new(),
sensitivity: None,
columns: vec![],
lineage: None,
freshness: None,
tags: vec![],
last_embedded_at: None,
embedding_stale_since: None,
embedding_refresh_policy: None,
};
let newer_no_rows = DatasetManifest {
id: DatasetId::new(),
updated_at: now, // newer, but null row_count
row_count: None,
..old_with_rows.clone()
};
let oldest = DatasetManifest {
id: DatasetId::new(),
updated_at: now - chrono::Duration::hours(5),
row_count: None,
..old_with_rows.clone()
};
let kept_id = old_with_rows.id.clone();
let loser_a = newer_no_rows.id.clone();
let loser_b = oldest.id.clone();
{
let mut d = reg.datasets.write().await;
d.insert(old_with_rows.id.clone(), old_with_rows);
d.insert(newer_no_rows.id.clone(), newer_no_rows);
d.insert(oldest.id.clone(), oldest);
}
assert_eq!(reg.list().await.len(), 3);
let report = reg.dedupe_by_name().await;
assert_eq!(report.groups, 1);
assert_eq!(report.removed, 2);
assert_eq!(report.errors.len(), 0);
assert_eq!(report.kept.len(), 1);
assert_eq!(report.kept[0].name, "dupes");
assert_eq!(report.kept[0].kept_id, kept_id.to_string(),
"should keep the manifest with non-null row_count");
let remaining = reg.list().await;
assert_eq!(remaining.len(), 1);
assert_eq!(remaining[0].id, kept_id);
assert!(reg.get(&loser_a).await.is_none());
assert!(reg.get(&loser_b).await.is_none());
}
}
crates/catalogd/src/service.rs
+35 -2
View File
@@ -17,11 +17,12 @@ pub fn router(registry: Registry) -> Router {
.route("/datasets", post(create_dataset)) .route("/datasets", post(create_dataset))
.route("/datasets", get(list_datasets)) .route("/datasets", get(list_datasets))
.route("/datasets/{id}", get(get_dataset)) .route("/datasets/{id}", get(get_dataset))
.route("/datasets/by-name/{name}", get(get_dataset_by_name)) .route("/datasets/by-name/{name}", get(get_dataset_by_name).delete(delete_dataset_by_name))
.route("/datasets/by-name/{name}/metadata", post(update_metadata)) .route("/datasets/by-name/{name}/metadata", post(update_metadata))
.route("/datasets/by-name/{name}/resync", post(resync_dataset)) .route("/datasets/by-name/{name}/resync", post(resync_dataset))
.route("/resync-missing", post(resync_all_missing)) .route("/resync-missing", post(resync_all_missing))
.route("/migrate-buckets", post(migrate_buckets)) .route("/migrate-buckets", post(migrate_buckets))
.route("/dedupe", post(dedupe_by_name))
// Phase D: AI-safe views // Phase D: AI-safe views
.route("/views", post(create_view).get(list_views)) .route("/views", post(create_view).get(list_views))
.route("/views/{name}", get(get_view).delete(delete_view)) .route("/views/{name}", get(get_view).delete(delete_view))
@@ -121,7 +122,14 @@ async fn create_dataset(
let resp = DatasetResponse::from(&manifest); let resp = DatasetResponse::from(&manifest);
Ok((StatusCode::CREATED, Json(resp))) Ok((StatusCode::CREATED, Json(resp)))
} }
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)), Err(e) => {
let status = if e.contains("different schema") {
StatusCode::CONFLICT
} else {
StatusCode::INTERNAL_SERVER_ERROR
};
Err((status, e))
}
} }
} }
@@ -153,6 +161,25 @@ async fn get_dataset_by_name(
} }
} }
/// Remove a dataset manifest by name. Metadata only — parquet files,
/// vector indexes, and tombstones are NOT cascade-deleted. See
/// `Registry::delete_dataset` for the full scope.
async fn delete_dataset_by_name(
State(registry): State<Registry>,
Path(name): Path<String>,
) -> impl IntoResponse {
match registry.delete_dataset(&name).await {
Ok(removed) => {
let body = serde_json::json!({ "name": name, "manifests_removed": removed });
Ok(Json(body))
}
Err(e) if e.starts_with("dataset not found") => {
Err((StatusCode::NOT_FOUND, e))
}
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
}
}
async fn update_metadata( async fn update_metadata(
State(registry): State<Registry>, State(registry): State<Registry>,
Path(name): Path<String>, Path(name): Path<String>,
@@ -214,6 +241,12 @@ async fn migrate_buckets(State(registry): State<Registry>) -> impl IntoResponse
} }
} }
/// Collapse duplicate manifests by `name` — see `Registry::dedupe_by_name`.
/// Safe to run repeatedly; single-manifest datasets are untouched.
async fn dedupe_by_name(State(registry): State<Registry>) -> impl IntoResponse {
Json(registry.dedupe_by_name().await)
}
// --- Phase D: AI-safe views --- // --- Phase D: AI-safe views ---
#[derive(Deserialize)] #[derive(Deserialize)]
crates/gateway/Cargo.toml
+1
View File
@@ -28,3 +28,4 @@ opentelemetry-stdout = { workspace = true }
tracing-opentelemetry = { workspace = true } tracing-opentelemetry = { workspace = true }
arrow = { workspace = true } arrow = { workspace = true }
chrono = { workspace = true } chrono = { workspace = true }
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
crates/gateway/src/main.rs
+42
View File
@@ -3,6 +3,7 @@ mod access_service;
mod auth; mod auth;
mod observability; mod observability;
mod tools; mod tools;
mod v1;
use axum::{Router, extract::DefaultBodyLimit, routing::get}; use axum::{Router, extract::DefaultBodyLimit, routing::get};
use proto::lakehouse::catalog_service_server::CatalogServiceServer; use proto::lakehouse::catalog_service_server::CatalogServiceServer;
@@ -86,6 +87,11 @@ async fn main() {
// primary. Journal / registry resolve per-call via the bucket registry. // primary. Journal / registry resolve per-call via the bucket registry.
let tj = vectord::trial::TrialJournal::new(bucket_registry.clone(), index_reg.clone()); let tj = vectord::trial::TrialJournal::new(bucket_registry.clone(), index_reg.clone());
let pr = vectord::promotion::PromotionRegistry::new(bucket_registry.clone(), index_reg.clone()); let pr = vectord::promotion::PromotionRegistry::new(bucket_registry.clone(), index_reg.clone());
let hs = vectord::harness::HarnessStore::new(bucket_registry.clone(), index_reg.clone());
// Phase 19: playbook memory. Load cached state; empty-on-miss is fine,
// operators call POST /vectors/playbook_memory/rebuild to populate.
let pbm = vectord::playbook_memory::PlaybookMemory::new(store.clone());
let _ = pbm.load_from_storage().await;
// Phase 16.2: spawn the autotune agent. When config.agent.enabled=false // Phase 16.2: spawn the autotune agent. When config.agent.enabled=false
// this returns a handle that drops triggers silently — no surprise load. // this returns a handle that drops triggers silently — no surprise load.
@@ -107,6 +113,7 @@ async fn main() {
embedding_cache: emb_cache.clone(), embedding_cache: emb_cache.clone(),
trial_journal: tj.clone(), trial_journal: tj.clone(),
promotion_registry: pr.clone(), promotion_registry: pr.clone(),
harness_store: hs.clone(),
}, },
); );
@@ -155,6 +162,7 @@ async fn main() {
hnsw_store: hnsw, hnsw_store: hnsw,
embedding_cache: emb_cache, embedding_cache: emb_cache,
trial_journal: tj, trial_journal: tj,
harness_store: hs,
catalog: registry.clone(), catalog: registry.clone(),
promotion_registry: pr, promotion_registry: pr,
agent_handle, agent_handle,
@@ -163,6 +171,8 @@ async fn main() {
lance: vectord::lance_backend::LanceRegistry::new( lance: vectord::lance_backend::LanceRegistry::new(
bucket_registry.clone(), index_reg.clone(), bucket_registry.clone(), index_reg.clone(),
), ),
playbook_memory: pbm,
embed_semaphore: std::sync::Arc::new(tokio::sync::Semaphore::new(1)),
})) }))
.nest("/workspaces", queryd::workspace_service::router(workspace_mgr)) .nest("/workspaces", queryd::workspace_service::router(workspace_mgr))
.nest("/journal", journald::service::router(journal)) .nest("/journal", journald::service::router(journal))
@@ -174,6 +184,38 @@ async fn main() {
registry: tool_reg, registry: tool_reg,
query_fn: tools::QueryExecutor::new(engine.clone()), query_fn: tools::QueryExecutor::new(engine.clone()),
} }
}))
// Phase 38 — Universal API skeleton. Thin OpenAI-compatible
// surface over the existing aibridge → Ollama path. Future
// phases add provider adapters (39), routing engine (40),
// session state (41), etc. All without changing this mount.
.nest("/v1", v1::router(v1::V1State {
ai_client: ai_client.clone(),
usage: std::sync::Arc::new(tokio::sync::RwLock::new(v1::Usage::default())),
// Phase 39 first slice — Ollama Cloud adapter. Key resolved
// from OLLAMA_CLOUD_KEY env, then /root/llm_team_config.json,
// then OLLAMA_CLOUD_API_KEY env. None = cloud routes 503.
ollama_cloud_key: {
let k = v1::ollama_cloud::resolve_cloud_key();
if k.is_some() {
tracing::info!("v1: Ollama Cloud key loaded — /v1/chat provider=ollama_cloud enabled");
} else {
tracing::warn!("v1: no Ollama Cloud key in env or /root/llm_team_config.json — cloud routes will 503");
}
k
},
// Phase 40 early deliverable — Langfuse trace emitter.
// Defaults match mcp-server/tracing.ts conventions so
// gateway traces land in the same staffing project.
langfuse: {
let c = v1::langfuse_trace::LangfuseClient::from_env_or_defaults();
if c.is_some() {
tracing::info!("v1: Langfuse tracing enabled — /v1/chat calls will appear at localhost:3001");
} else {
tracing::warn!("v1: Langfuse keys missing — /v1/chat calls will not be traced");
}
c
},
})); }));
// Auth middleware (if enabled) // Auth middleware (if enabled)
crates/gateway/src/tools/registry.rs
+1 -1
View File
@@ -103,7 +103,7 @@ impl ToolRegistry {
ParamDef { name: "limit".into(), param_type: "integer".into(), required: false, description: "Max results".into(), default: Some(serde_json::json!(20)) }, ParamDef { name: "limit".into(), param_type: "integer".into(), required: false, description: "Max results".into(), default: Some(serde_json::json!(20)) },
], ],
returns: "List of candidates with id, name, phone, email, skills, experience".into(), returns: "List of candidates with id, name, phone, email, skills, experience".into(),
sql_template: "SELECT candidate_id, first_name, last_name, phone, email, city, state, zip, vertical, skills, years_experience FROM candidates WHERE 1=1 {skills_filter} {city_filter} {state_filter} {years_filter} {status_filter} ORDER BY years_experience DESC LIMIT {limit}".into(), sql_template: "SELECT candidate_id, first_name, last_name, phone, email, city, state, skills, years_experience, status FROM candidates WHERE 1=1 {skills_filter} {city_filter} {state_filter} {years_filter} {status_filter} ORDER BY years_experience DESC LIMIT {limit}".into(),
category: "candidates".into(), category: "candidates".into(),
}, },
ToolDef { ToolDef {
crates/gateway/src/v1/langfuse_trace.rs
+226
View File
@@ -0,0 +1,226 @@
//! Phase 40 (early deliverable) — Langfuse tracing from the Rust gateway.
//!
//! Langfuse is already running at `http://localhost:3001` with the
//! staffing project — `mcp-server/tracing.ts` has been emitting traces
//! for months. This file bridges the gap: every `/v1/chat` call now
//! appears in the same Langfuse project, alongside traces from the
//! Bun side.
//!
//! Design:
//! - Fire-and-forget — emit spawns a tokio task. Response to client
//! never waits on Langfuse, never fails because of it. If Langfuse
//! is down or slow, we silently drop traces (and `tracing::warn!`).
//! - Two events per call — `trace-create` + `generation-create`. The
//! generation nests under the trace so Langfuse UI shows the whole
//! thing as one clickable row.
//! - No SDK — plain HTTP POST to `/api/public/ingestion` with Basic
//! auth. Keeps the Rust dependency graph small.
use serde::Serialize;
use std::sync::Arc;
use std::time::Duration;
use super::Message;
const INGESTION_PATH: &str = "/api/public/ingestion";
const EMIT_TIMEOUT_SECS: u64 = 5;
#[derive(Clone)]
pub struct LangfuseClient {
inner: Arc<Inner>,
}
struct Inner {
base_url: String,
public_key: String,
secret_key: String,
http: reqwest::Client,
}
impl LangfuseClient {
pub fn new(base_url: String, public_key: String, secret_key: String) -> Self {
let http = reqwest::Client::builder()
.timeout(Duration::from_secs(EMIT_TIMEOUT_SECS))
.build()
.expect("langfuse client init");
Self { inner: Arc::new(Inner { base_url, public_key, secret_key, http }) }
}
/// Resolve a LangfuseClient from the same conventions the TS side
/// already uses. Returns None if keys are missing — caller logs a
/// warning and omits tracing (non-fatal).
///
/// Sources checked in order:
/// 1. LANGFUSE_URL + LANGFUSE_PUBLIC_KEY + LANGFUSE_SECRET_KEY env
/// 2. /root/llm_team_config.json (if it carries langfuse keys)
/// 3. Hardcoded staffing-project defaults matching mcp-server
/// (pk-lf-staffing / sk-lf-staffing-secret / localhost:3001)
pub fn from_env_or_defaults() -> Option<Self> {
let base = std::env::var("LANGFUSE_URL")
.unwrap_or_else(|_| "http://localhost:3001".to_string());
let pk = std::env::var("LANGFUSE_PUBLIC_KEY")
.unwrap_or_else(|_| "pk-lf-staffing".to_string());
let sk = std::env::var("LANGFUSE_SECRET_KEY")
.unwrap_or_else(|_| "sk-lf-staffing-secret".to_string());
if pk.trim().is_empty() || sk.trim().is_empty() { return None; }
Some(Self::new(base, pk, sk))
}
/// Fire-and-forget emit. Never blocks, never errors upward.
pub fn emit_chat(&self, ev: ChatTrace) {
let this = self.clone();
tokio::spawn(async move {
if let Err(e) = this.emit_chat_inner(ev).await {
tracing::warn!(target: "v1.langfuse", "trace drop: {e}");
}
});
}
async fn emit_chat_inner(&self, ev: ChatTrace) -> Result<(), String> {
let trace_id = uuid_v7_like();
let gen_id = uuid_v7_like();
let trace_ts = ev.start_time.clone();
let batch = IngestionBatch {
batch: vec![
IngestionEvent {
id: uuid_v7_like(),
timestamp: trace_ts.clone(),
kind: "trace-create",
body: serde_json::json!({
"id": trace_id,
"name": format!("v1.chat:{}", ev.provider),
"input": serde_json::json!({
"model": ev.model,
"messages": ev.input,
}),
"metadata": serde_json::json!({
"provider": ev.provider,
"think": ev.think,
}),
}),
},
IngestionEvent {
id: uuid_v7_like(),
timestamp: ev.end_time.clone(),
kind: "generation-create",
body: serde_json::json!({
"id": gen_id,
"traceId": trace_id,
"name": "chat",
"model": ev.model,
"modelParameters": serde_json::json!({
"temperature": ev.temperature,
"max_tokens": ev.max_tokens,
"think": ev.think,
}),
"input": ev.input,
"output": ev.output,
"usage": serde_json::json!({
"input": ev.prompt_tokens,
"output": ev.completion_tokens,
"total": ev.prompt_tokens + ev.completion_tokens,
"unit": "TOKENS",
}),
"startTime": ev.start_time,
"endTime": ev.end_time,
"metadata": serde_json::json!({
"provider": ev.provider,
"latency_ms": ev.latency_ms,
}),
}),
},
],
};
let url = format!("{}{}", self.inner.base_url.trim_end_matches('/'), INGESTION_PATH);
let resp = self.inner.http
.post(url)
.basic_auth(&self.inner.public_key, Some(&self.inner.secret_key))
.json(&batch)
.send()
.await
.map_err(|e| format!("POST failed: {e}"))?;
if !resp.status().is_success() {
return Err(format!("{}: {}", resp.status(), resp.text().await.unwrap_or_default()));
}
Ok(())
}
}
/// Everything the v1.chat handler collects for one completed call.
pub struct ChatTrace {
pub provider: String,
pub model: String,
pub input: Vec<Message>,
pub output: String,
pub prompt_tokens: u32,
pub completion_tokens: u32,
pub temperature: Option<f64>,
pub max_tokens: Option<u32>,
pub think: Option<bool>,
pub start_time: String,
pub end_time: String,
pub latency_ms: u64,
}
#[derive(Serialize)]
struct IngestionBatch {
batch: Vec<IngestionEvent>,
}
#[derive(Serialize)]
struct IngestionEvent {
id: String,
timestamp: String,
#[serde(rename = "type")]
kind: &'static str,
body: serde_json::Value,
}
/// UUIDv7-ish identifier — not strictly UUIDv7 but time-ordered and
/// unique enough for event deduplication. Avoids a uuid crate dep.
fn uuid_v7_like() -> String {
let ts = chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0);
let rand = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.subsec_nanos())
.unwrap_or(0);
format!("{:016x}-{:08x}", ts, rand)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn from_env_or_defaults_returns_some_when_defaults_present() {
// Defaults are hardcoded non-empty — should always produce Some.
let c = LangfuseClient::from_env_or_defaults();
assert!(c.is_some());
}
#[test]
fn uuid_v7_like_produces_distinct_ids() {
let a = uuid_v7_like();
// Different nanosecond timestamps — but they might collide on
// back-to-back calls. Loop a bit to ensure uniqueness surface.
let b = uuid_v7_like();
let c = uuid_v7_like();
assert!(a != b || b != c, "three consecutive ids should have at least one distinct pair");
assert!(a.contains('-'));
}
#[test]
fn ingestion_batch_serializes_with_rename() {
let ev = IngestionEvent {
id: "e1".into(),
timestamp: "2026-04-22T00:00:00Z".into(),
kind: "trace-create",
body: serde_json::json!({"name": "t"}),
};
let json = serde_json::to_string(&IngestionBatch { batch: vec![ev] }).unwrap();
assert!(json.contains("\"type\":\"trace-create\""));
assert!(json.contains("\"batch\":["));
}
}
crates/gateway/src/v1/mod.rs
+256
View File
@@ -0,0 +1,256 @@
//! Phase 38 — Universal API skeleton (`/v1/*`).
//!
//! OpenAI-compatible shape on top of the existing aibridge → Ollama
//! path. This is the thin slice: single provider, stateless, no
//! streaming. Phase 39 replaces the direct Ollama call with a
//! `ProviderAdapter` trait dispatch; Phase 40 adds routing + fallback.
//!
//! The shape matches OpenAI's `/v1/chat/completions` closely enough
//! that clients using openai-compatible SDKs can point at us with the
//! URL swap alone. We keep the endpoint path `/v1/chat` (not
//! `/v1/chat/completions`) because our PRD declares the terser form;
//! adding the alias is one line in Phase 39 when it matters.
pub mod ollama;
pub mod ollama_cloud;
pub mod langfuse_trace;
use axum::{
Router,
extract::State,
http::StatusCode,
response::IntoResponse,
routing::{get, post},
Json,
};
use serde::{Deserialize, Serialize};
use std::sync::Arc;
use tokio::sync::RwLock;
#[derive(Clone)]
pub struct V1State {
pub ai_client: aibridge::client::AiClient,
pub usage: Arc<RwLock<Usage>>,
/// Ollama Cloud bearer token. Loaded at startup via
/// `ollama_cloud::resolve_cloud_key()`. None = cloud routes 503.
pub ollama_cloud_key: Option<String>,
/// Phase 40 early deliverable — Langfuse client. None = tracing
/// disabled (keys missing or container unreachable). Traces are
/// fire-and-forget: never block the response path.
pub langfuse: Option<langfuse_trace::LangfuseClient>,
}
#[derive(Default, Clone, Serialize)]
pub struct Usage {
pub requests: u64,
pub prompt_tokens: u64,
pub completion_tokens: u64,
pub total_tokens: u64,
}
pub fn router(state: V1State) -> Router {
Router::new()
.route("/chat", post(chat))
.route("/usage", get(usage))
.route("/sessions", get(sessions))
.with_state(state)
}
// -- Shared types (OpenAI-compatible) --
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct Message {
pub role: String,
pub content: String,
}
#[derive(Deserialize, Debug)]
pub struct ChatRequest {
pub model: String,
pub messages: Vec<Message>,
#[serde(default)]
pub temperature: Option<f64>,
#[serde(default)]
pub max_tokens: Option<u32>,
/// Accepted for shape-compat but ignored in the thin slice —
/// Phase 38 returns non-streaming even when the client asked for it.
/// Phase 39+ wires real streaming.
#[serde(default)]
pub stream: Option<bool>,
/// Non-OpenAI extension. Passes through to the provider's thinking
/// toggle. Default: **false** — hot-path discipline for thinking
/// models (qwen3.5, qwen3, gpt-oss) that otherwise burn the token
/// budget on hidden reasoning before visible output starts,
/// producing empty responses. Set true explicitly when calling an
/// overseer / reasoning-heavy path.
#[serde(default)]
pub think: Option<bool>,
/// Non-OpenAI extension. Selects the provider adapter. Accepted:
/// - None / "ollama" / "local" → local Ollama via sidecar (default)
/// - "ollama_cloud" / "cloud" → Ollama Cloud direct HTTPS
/// Phase 40 adds a routing engine that picks this automatically
/// from the model name; Phase 38/39 requires explicit selection.
#[serde(default)]
pub provider: Option<String>,
}
#[derive(Serialize)]
pub struct ChatResponse {
pub id: String,
pub object: &'static str,
pub created: i64,
pub model: String,
pub choices: Vec<Choice>,
pub usage: UsageBlock,
}
#[derive(Serialize)]
pub struct Choice {
pub index: u32,
pub message: Message,
pub finish_reason: String,
}
#[derive(Serialize, Deserialize, Clone)]
pub struct UsageBlock {
pub prompt_tokens: u32,
pub completion_tokens: u32,
pub total_tokens: u32,
}
// -- Handlers --
async fn chat(
State(state): State<V1State>,
Json(req): Json<ChatRequest>,
) -> Result<Json<ChatResponse>, (StatusCode, String)> {
if req.messages.is_empty() {
return Err((StatusCode::BAD_REQUEST, "messages must be non-empty".into()));
}
if req.stream.unwrap_or(false) {
tracing::warn!("/v1/chat: stream=true requested but Phase 38 returns non-streaming");
}
let provider = req.provider.as_deref().unwrap_or("ollama").to_ascii_lowercase();
let start_time = chrono::Utc::now();
let start_instant = std::time::Instant::now();
let resp = match provider.as_str() {
"ollama" | "local" | "" => ollama::chat(&state.ai_client, &req)
.await
.map_err(|e| (StatusCode::BAD_GATEWAY, format!("ollama local: {e}")))?,
"ollama_cloud" | "cloud" => {
let key = state.ollama_cloud_key.as_deref().ok_or((
StatusCode::SERVICE_UNAVAILABLE,
"OLLAMA_CLOUD_KEY not configured — set env or populate /root/llm_team_config.json".to_string(),
))?;
ollama_cloud::chat(key, &req)
.await
.map_err(|e| (StatusCode::BAD_GATEWAY, format!("ollama cloud: {e}")))?
}
other => {
return Err((
StatusCode::BAD_REQUEST,
format!("unknown provider '{other}' — supported: ollama, ollama_cloud"),
));
}
};
let end_time = chrono::Utc::now();
let latency_ms = start_instant.elapsed().as_millis() as u64;
// Phase 40 — emit Langfuse trace. Fire-and-forget: the clone is
// cheap (Arc inside), the tokio::spawn never blocks us, a dead
// Langfuse just logs a warn. Client-visible response latency is
// untouched.
if let Some(lf) = &state.langfuse {
let output = resp.choices.first()
.map(|c| c.message.content.clone())
.unwrap_or_default();
lf.emit_chat(langfuse_trace::ChatTrace {
provider: provider.clone(),
model: resp.model.clone(),
input: req.messages.clone(),
output,
prompt_tokens: resp.usage.prompt_tokens,
completion_tokens: resp.usage.completion_tokens,
temperature: req.temperature,
max_tokens: req.max_tokens,
think: req.think,
start_time: start_time.to_rfc3339(),
end_time: end_time.to_rfc3339(),
latency_ms,
});
}
{
let mut u = state.usage.write().await;
u.requests += 1;
u.prompt_tokens += resp.usage.prompt_tokens as u64;
u.completion_tokens += resp.usage.completion_tokens as u64;
u.total_tokens += resp.usage.total_tokens as u64;
}
Ok(Json(resp))
}
async fn usage(State(state): State<V1State>) -> impl IntoResponse {
let snapshot = state.usage.read().await.clone();
Json(snapshot)
}
// Phase 38 is stateless — no session persistence yet. Return an empty
// list in OpenAI-ish shape so clients that probe this endpoint don't
// 404. Real session state lands in Phase 41 with the profile-system
// expansion.
async fn sessions() -> impl IntoResponse {
Json(serde_json::json!({
"data": [],
"object": "list",
"note": "Phase 38: stateless. Session state lands in Phase 41.",
}))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn chat_request_parses_openai_shape() {
let raw = r#"{
"model": "qwen3.5:latest",
"messages": [
{"role": "system", "content": "You are helpful."},
{"role": "user", "content": "Hi"}
],
"temperature": 0.2,
"max_tokens": 100
}"#;
let r: ChatRequest = serde_json::from_str(raw).unwrap();
assert_eq!(r.model, "qwen3.5:latest");
assert_eq!(r.messages.len(), 2);
assert_eq!(r.messages[0].role, "system");
assert_eq!(r.messages[1].content, "Hi");
assert_eq!(r.temperature, Some(0.2));
assert_eq!(r.max_tokens, Some(100));
}
#[test]
fn chat_request_accepts_minimal() {
let raw = r#"{
"model": "any",
"messages": [{"role": "user", "content": "hi"}]
}"#;
let r: ChatRequest = serde_json::from_str(raw).unwrap();
assert_eq!(r.temperature, None);
assert_eq!(r.max_tokens, None);
assert_eq!(r.stream, None);
}
#[test]
fn usage_counter_default_is_zero() {
let u = Usage::default();
assert_eq!(u.requests, 0);
assert_eq!(u.total_tokens, 0);
}
}
crates/gateway/src/v1/ollama.rs
+160
View File
@@ -0,0 +1,160 @@
//! Phase 38 — Ollama shape adapter.
//!
//! Translates `/v1/chat` (OpenAI-compatible) requests into the
//! existing aibridge `GenerateRequest` shape, and the `GenerateResponse`
//! back into an OpenAI-compatible `ChatResponse`. This is a bridge,
//! not a new client — aibridge + the Python sidecar stay as-is.
//!
//! Phase 39 replaces this direct call with a `ProviderAdapter` trait
//! dispatch so the same `/v1/chat` handler routes to any provider.
use aibridge::client::{AiClient, GenerateRequest};
use super::{ChatRequest, ChatResponse, Choice, Message, UsageBlock};
pub async fn chat(client: &AiClient, req: &ChatRequest) -> Result<ChatResponse, String> {
let (system, prompt) = flatten_messages(&req.messages);
let gen_req = GenerateRequest {
prompt,
model: Some(req.model.clone()),
system: if system.is_empty() { None } else { Some(system) },
temperature: req.temperature,
max_tokens: req.max_tokens,
// Phase 38 default: think=false. Hot-path discipline for
// thinking models (qwen3.5, qwen3, gpt-oss) which otherwise
// burn the max_tokens budget on hidden reasoning before any
// visible output, producing empty responses. Callers that
// actually want reasoning (overseers, T3+ tiers) opt in via
// the `think: true` extension field. Phase 40 routing engine
// flips this per task class.
think: Some(req.think.unwrap_or(false)),
};
let t0 = std::time::Instant::now();
let resp = client.generate(gen_req).await.map_err(|e| e.to_string())?;
let latency_ms = t0.elapsed().as_millis();
// Prefer sidecar-reported token counts when present. Fall back to
// chars/4 estimate (biased safe ~15%, matches Phase 21 convention
// in crates/aibridge/src/context.rs::estimate_tokens).
let prompt_tokens = resp.tokens_evaluated
.map(|n| n as u32)
.unwrap_or_else(|| estimate_prompt_tokens(&req.messages));
let completion_tokens = resp.tokens_generated
.map(|n| n as u32)
.unwrap_or_else(|| ((resp.text.chars().count() + 3) / 4) as u32);
tracing::info!(
target: "v1.chat",
model = %req.model,
prompt_tokens,
completion_tokens,
latency_ms = latency_ms as u64,
"ollama chat completed",
);
Ok(ChatResponse {
id: format!("chatcmpl-{}", chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0)),
object: "chat.completion",
created: chrono::Utc::now().timestamp(),
model: resp.model,
choices: vec![Choice {
index: 0,
message: Message {
role: "assistant".into(),
content: resp.text,
},
finish_reason: "stop".into(),
}],
usage: UsageBlock {
prompt_tokens,
completion_tokens,
total_tokens: prompt_tokens + completion_tokens,
},
})
}
/// Public re-export of the flattener so sibling adapters (Ollama Cloud,
/// future OpenRouter) can reuse the same shape collapse without
/// duplicating the logic. Keeps `(system, prompt)` format consistent
/// across providers.
pub fn flatten_messages_public(messages: &[Message]) -> (String, String) {
flatten_messages(messages)
}
/// Collapse a message array into (system, prompt). Multiple system
/// messages concatenate with a newline — matches OpenAI's documented
/// behavior. Non-system messages become role-labeled turns.
fn flatten_messages(messages: &[Message]) -> (String, String) {
let mut system = String::new();
let mut prompt = String::new();
for m in messages {
if m.role == "system" {
if !system.is_empty() { system.push('\n'); }
system.push_str(&m.content);
} else {
prompt.push_str(&m.role);
prompt.push_str(": ");
prompt.push_str(&m.content);
prompt.push_str("\n\n");
}
}
prompt.push_str("assistant:");
(system, prompt)
}
fn estimate_prompt_tokens(messages: &[Message]) -> u32 {
let chars: usize = messages.iter().map(|m| m.content.chars().count()).sum();
((chars + 3) / 4) as u32
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn flatten_separates_system_from_turns() {
let msgs = vec![
Message { role: "system".into(), content: "Rules here.".into() },
Message { role: "user".into(), content: "Q1".into() },
Message { role: "assistant".into(), content: "A1".into() },
Message { role: "user".into(), content: "Q2".into() },
];
let (system, prompt) = flatten_messages(&msgs);
assert_eq!(system, "Rules here.");
assert!(prompt.contains("user: Q1"));
assert!(prompt.contains("assistant: A1"));
assert!(prompt.contains("user: Q2"));
assert!(prompt.trim_end().ends_with("assistant:"));
}
#[test]
fn flatten_concatenates_multiple_system_messages() {
let msgs = vec![
Message { role: "system".into(), content: "First.".into() },
Message { role: "system".into(), content: "Second.".into() },
Message { role: "user".into(), content: "Hi".into() },
];
let (system, _) = flatten_messages(&msgs);
assert_eq!(system, "First.\nSecond.");
}
#[test]
fn flatten_with_no_system_returns_empty_system() {
let msgs = vec![Message { role: "user".into(), content: "hi".into() }];
let (system, prompt) = flatten_messages(&msgs);
assert!(system.is_empty());
assert!(prompt.contains("user: hi"));
}
#[test]
fn estimate_tokens_chars_div_4_ceiling() {
let msgs = vec![Message { role: "user".into(), content: "abcdefgh".into() }];
// 8 chars / 4 = 2, with ceiling → 2
assert_eq!(estimate_prompt_tokens(&msgs), 2);
let msgs2 = vec![Message { role: "user".into(), content: "abcdefghi".into() }];
// 9 chars → (9+3)/4 = 3 (ceiling)
assert_eq!(estimate_prompt_tokens(&msgs2), 3);
}
}
crates/gateway/src/v1/ollama_cloud.rs
+189
View File
@@ -0,0 +1,189 @@
//! Phase 39 (first slice) — Ollama Cloud adapter.
//!
//! Direct HTTPS call to `https://ollama.com/api/generate` with Bearer
//! auth. Mirrors the `tests/multi-agent/agent.ts::generateCloud` pattern
//! exactly (same endpoint, same body shape, same Bearer header), so
//! cloud calls from Rust behave identically to the TS hot path.
//!
//! Key sourcing priority:
//! 1. Env var `OLLAMA_CLOUD_KEY` (matches agent.ts convention)
//! 2. `/root/llm_team_config.json` → providers.ollama_cloud.api_key
//! 3. Env var `OLLAMA_CLOUD_API_KEY` (LLM Team UI convention)
//!
//! First hit wins. Key is loaded once at gateway startup by
//! `resolve_cloud_key()` and stored on `V1State.ollama_cloud_key`.
use std::time::Duration;
use serde::{Deserialize, Serialize};
use super::{ChatRequest, ChatResponse, Choice, Message, UsageBlock};
const CLOUD_BASE_URL: &str = "https://ollama.com";
const CLOUD_TIMEOUT_SECS: u64 = 180;
/// Read the Ollama Cloud key from the three sanctioned sources. Returns
/// None if none is set — callers must 503 rather than attempt a call.
pub fn resolve_cloud_key() -> Option<String> {
if let Ok(k) = std::env::var("OLLAMA_CLOUD_KEY") {
if !k.trim().is_empty() { return Some(k.trim().to_string()); }
}
if let Ok(raw) = std::fs::read_to_string("/root/llm_team_config.json") {
if let Ok(v) = serde_json::from_str::<serde_json::Value>(&raw) {
if let Some(k) = v.pointer("/providers/ollama_cloud/api_key").and_then(|x| x.as_str()) {
if !k.trim().is_empty() { return Some(k.trim().to_string()); }
}
}
}
if let Ok(k) = std::env::var("OLLAMA_CLOUD_API_KEY") {
if !k.trim().is_empty() { return Some(k.trim().to_string()); }
}
None
}
pub async fn chat(
key: &str,
req: &ChatRequest,
) -> Result<ChatResponse, String> {
let (system, prompt) = super::ollama::flatten_messages_public(&req.messages);
let body = CloudGenerateBody {
model: req.model.clone(),
prompt,
system: if system.is_empty() { None } else { Some(system) },
stream: false,
think: Some(req.think.unwrap_or(false)),
options: CloudOptions {
// Thinking cloud models need headroom — floor 400 to give
// qwen3.5:397b / gpt-oss:120b reasoning room. Matches
// agent.ts `Math.max(opts.max_tokens ?? 800, 400)` policy.
num_predict: std::cmp::max(req.max_tokens.unwrap_or(800), 400),
temperature: req.temperature.unwrap_or(0.3),
},
};
let client = reqwest::Client::builder()
.timeout(Duration::from_secs(CLOUD_TIMEOUT_SECS))
.build()
.map_err(|e| format!("build client: {e}"))?;
let t0 = std::time::Instant::now();
let resp = client
.post(format!("{}/api/generate", CLOUD_BASE_URL))
.bearer_auth(key)
.json(&body)
.send()
.await
.map_err(|e| format!("ollama.com unreachable: {e}"))?;
let status = resp.status();
if !status.is_success() {
let body = resp.text().await.unwrap_or_else(|_| "?".into());
return Err(format!("ollama.com {}: {}", status, body));
}
let parsed: CloudGenerateResponse = resp.json().await
.map_err(|e| format!("invalid cloud response: {e}"))?;
let latency_ms = t0.elapsed().as_millis();
let text = parsed.response.unwrap_or_default();
let prompt_tokens = parsed.prompt_eval_count.unwrap_or_else(|| {
let chars: usize = req.messages.iter().map(|m| m.content.chars().count()).sum();
((chars + 3) / 4) as u32
});
let completion_tokens = parsed.eval_count.unwrap_or_else(|| {
((text.chars().count() + 3) / 4) as u32
});
tracing::info!(
target: "v1.chat",
provider = "ollama_cloud",
model = %req.model,
prompt_tokens,
completion_tokens,
latency_ms = latency_ms as u64,
"cloud chat completed",
);
Ok(ChatResponse {
id: format!("chatcmpl-{}", chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0)),
object: "chat.completion",
created: chrono::Utc::now().timestamp(),
model: parsed.model.unwrap_or_else(|| req.model.clone()),
choices: vec![Choice {
index: 0,
message: Message { role: "assistant".into(), content: text },
finish_reason: "stop".into(),
}],
usage: UsageBlock {
prompt_tokens,
completion_tokens,
total_tokens: prompt_tokens + completion_tokens,
},
})
}
// -- Ollama /api/generate wire shapes --
#[derive(Serialize)]
struct CloudGenerateBody {
model: String,
prompt: String,
#[serde(skip_serializing_if = "Option::is_none")]
system: Option<String>,
stream: bool,
#[serde(skip_serializing_if = "Option::is_none")]
think: Option<bool>,
options: CloudOptions,
}
#[derive(Serialize)]
struct CloudOptions {
num_predict: u32,
temperature: f64,
}
#[derive(Deserialize)]
struct CloudGenerateResponse {
#[serde(default)]
response: Option<String>,
#[serde(default)]
model: Option<String>,
#[serde(default)]
prompt_eval_count: Option<u32>,
#[serde(default)]
eval_count: Option<u32>,
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn resolve_cloud_key_returns_none_when_no_sources_set() {
// Only check shape — we can't reliably unset env vars in a test
// that runs alongside others, and the file path is on disk.
// If all three sources are empty the function returns None; if
// any are set we expect Some. This just smoke-tests the call
// doesn't panic.
let _ = resolve_cloud_key();
}
#[test]
fn cloud_body_serializes_compact() {
let body = CloudGenerateBody {
model: "gpt-oss:120b".into(),
prompt: "user: hi\n\nassistant:".into(),
system: Some("Be terse.".into()),
stream: false,
think: Some(false),
options: CloudOptions { num_predict: 400, temperature: 0.3 },
};
let json = serde_json::to_string(&body).unwrap();
assert!(json.contains("\"model\":\"gpt-oss:120b\""));
assert!(json.contains("\"stream\":false"));
assert!(json.contains("\"num_predict\":400"));
assert!(json.contains("\"think\":false"));
assert!(json.contains("\"system\":\"Be terse.\""));
}
}
crates/ingestd/Cargo.toml
+1
View File
@@ -24,3 +24,4 @@ object_store = { workspace = true }
tokio-postgres = { workspace = true } tokio-postgres = { workspace = true }
mysql_async = { workspace = true } mysql_async = { workspace = true }
uuid = { workspace = true } uuid = { workspace = true }
croner = "3"
crates/ingestd/src/schedule.rs
+112 -13
View File
@@ -17,8 +17,8 @@
//! shouldn't pile up runs). Different schedules run concurrently. //! shouldn't pile up runs). Different schedules run concurrently.
//! //!
//! What's deliberately not in scope here: //! What's deliberately not in scope here:
//! - Cron expressions (the trigger enum has the variant but parsing is //! - Sub-minute cron precision. 5-field Unix cron is supported; seconds
//! stubbed). Intervals cover 90% of operational scheduling; cron is //! are pinned to 0. Intervals cover sub-minute cases; cron is
//! easy to bolt on later. //! easy to bolt on later.
//! - Backoff / retry policies. A failed run records the failure and //! - Backoff / retry policies. A failed run records the failure and
//! schedules `next_run_at` as if it succeeded — no exponential backoff. //! schedules `next_run_at` as if it succeeded — no exponential backoff.
@@ -85,8 +85,12 @@ impl ScheduleKind {
pub enum ScheduleTrigger { pub enum ScheduleTrigger {
/// Run every N seconds. Time anchor = compute_next_run_at decision. /// Run every N seconds. Time anchor = compute_next_run_at decision.
Interval { secs: u64 }, Interval { secs: u64 },
/// Cron expression — parsing not implemented yet. Defining the /// 5-field Unix cron expression: "min hour dom month dow".
/// variant now so the JSON shape is forward-compatible. /// Seconds are implicitly 0 (minute-granularity scheduling).
/// Examples: "15 14 * * *" = 14:15 UTC daily, "0 */6 * * *" = every
/// 6 hours on the hour, "0 9 * * 1-5" = 09:00 UTC weekdays.
/// Timezone: UTC always — the scheduler's clock is UTC and we don't
/// carry a per-schedule timezone field.
Cron { expr: String }, Cron { expr: String },
} }
@@ -162,22 +166,55 @@ impl ScheduleDef {
// Parse the expression, find the next match after `now`. Most // Parse the expression, find the next match after `now`. Most
// expressive but pulls in a cron parser dep. // expressive but pulls in a cron parser dep.
// //
// Cron is currently stubbed — falling back to "now + 1 hour" so a
// mistakenly-configured Cron schedule doesn't melt the GPU.
pub fn compute_next_run_at( pub fn compute_next_run_at(
trigger: &ScheduleTrigger, trigger: &ScheduleTrigger,
completed_at: DateTime<Utc>, completed_at: DateTime<Utc>,
_previous_next: DateTime<Utc>, _previous_next: DateTime<Utc>,
) -> DateTime<Utc> { ) -> DateTime<Utc> {
// TODO(J): pick a strategy. Starter = since-last-completion.
match trigger { match trigger {
ScheduleTrigger::Interval { secs } => { ScheduleTrigger::Interval { secs } => {
completed_at + Duration::seconds(*secs as i64) completed_at + Duration::seconds(*secs as i64)
} }
ScheduleTrigger::Cron { expr: _ } => { ScheduleTrigger::Cron { expr } => {
// Cron parsing not implemented — fall back to a safe 1h next_cron_fire(expr, completed_at)
// window so a bad config can't fire-loop the system. // Creation-time validation (see `validate_trigger`) rejects
completed_at + Duration::hours(1) // unparseable expressions, so this fallback is only reached
// if the schedule JSON was hand-edited on disk. One-hour
// window keeps a bad expr from fire-looping the system.
.unwrap_or(completed_at + Duration::hours(1))
}
}
}
/// Parse a Vixie/POSIX cron expression and return the next fire time
/// strictly after `after`. Accepts 5-field (`min hour dom month dow`) and
/// 6-field (with leading `sec`) forms natively — croner handles both.
/// Day-of-week follows Unix convention: 0=Sun, 1=Mon, …, 6=Sat (7=Sun as
/// synonym). Returns None if the expression fails to parse or produces no
/// upcoming match (which can happen for year-bounded patterns in the past).
fn next_cron_fire(expr: &str, after: DateTime<Utc>) -> Option<DateTime<Utc>> {
use std::str::FromStr;
let cron = croner::Cron::from_str(expr).ok()?;
cron.find_next_occurrence(&after, false).ok()
}
/// Reject a ScheduleTrigger whose content can't be interpreted — used by
/// the HTTP handlers to fail fast at create/patch time rather than falling
/// back silently at fire time. Interval triggers get a sanity gate
/// (no zero secs); cron triggers get a full parse through croner.
pub fn validate_trigger(trigger: &ScheduleTrigger) -> Result<(), String> {
match trigger {
ScheduleTrigger::Interval { secs } => {
if *secs == 0 {
return Err("interval secs must be > 0".into());
}
Ok(())
}
ScheduleTrigger::Cron { expr } => {
use std::str::FromStr;
croner::Cron::from_str(expr)
.map_err(|e| format!("invalid cron expression '{expr}': {e}"))?;
Ok(())
} }
} }
} }
@@ -551,6 +588,7 @@ fn redact_dsn(dsn: &str) -> String {
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
use chrono::Timelike;
fn mk(secs: u64) -> ScheduleDef { fn mk(secs: u64) -> ScheduleDef {
ScheduleDef { ScheduleDef {
@@ -600,12 +638,73 @@ mod tests {
} }
#[test] #[test]
fn cron_falls_back_to_one_hour() { fn cron_every_minute_fires_within_sixty_seconds() {
let trig = ScheduleTrigger::Cron { expr: "* * * * *".into() }; let trig = ScheduleTrigger::Cron { expr: "* * * * *".into() };
let now = Utc::now(); let now = Utc::now();
let next = compute_next_run_at(&trig, now, now); let next = compute_next_run_at(&trig, now, now);
let delta = next - now; let delta = next - now;
assert_eq!(delta.num_seconds(), 3600); assert!(delta.num_seconds() > 0 && delta.num_seconds() <= 60,
"expected next fire within 60s, got {}s", delta.num_seconds());
}
#[test]
fn cron_daily_at_1415_utc_is_within_24h() {
// 14:15 UTC daily — whether we're before or after 14:15 today, the
// next fire is at most 24h out.
let trig = ScheduleTrigger::Cron { expr: "15 14 * * *".into() };
let now: DateTime<Utc> = "2026-04-20T10:00:00Z".parse().unwrap();
let next = compute_next_run_at(&trig, now, now);
assert!(next > now);
assert!((next - now).num_hours() <= 24);
// And the minute-of-hour is 15.
assert_eq!(next.minute(), 15);
assert_eq!(next.hour(), 14);
}
#[test]
fn cron_weekday_skips_weekend() {
// 09:00 UTC weekdays only. On a Saturday, next fire is Monday.
let trig = ScheduleTrigger::Cron { expr: "0 9 * * 1-5".into() };
// 2026-04-18 is a Saturday.
let sat: DateTime<Utc> = "2026-04-18T10:00:00Z".parse().unwrap();
let next = compute_next_run_at(&trig, sat, sat);
// Monday 2026-04-20 at 09:00 UTC.
assert_eq!(next, "2026-04-20T09:00:00Z".parse::<DateTime<Utc>>().unwrap());
}
#[test]
fn cron_six_field_seconds_granularity() {
// 6-field (seconds included): fire every minute at 30s past the minute.
let trig = ScheduleTrigger::Cron { expr: "30 * * * * *".into() };
let now = Utc::now();
let next = compute_next_run_at(&trig, now, now);
assert_eq!(next.second(), 30);
}
#[test]
fn validate_rejects_bad_cron() {
let bad = ScheduleTrigger::Cron { expr: "not a cron".into() };
let err = validate_trigger(&bad).unwrap_err();
assert!(err.contains("cron"), "error should mention cron, got: {err}");
}
#[test]
fn validate_rejects_wrong_field_count() {
// 4 fields — neither 5 nor 6.
let bad = ScheduleTrigger::Cron { expr: "* * * *".into() };
assert!(validate_trigger(&bad).is_err());
}
#[test]
fn validate_rejects_zero_interval() {
let bad = ScheduleTrigger::Interval { secs: 0 };
assert!(validate_trigger(&bad).is_err());
}
#[test]
fn validate_accepts_good_cron() {
let good = ScheduleTrigger::Cron { expr: "0 */6 * * *".into() };
assert!(validate_trigger(&good).is_ok());
} }
#[test] #[test]
crates/ingestd/src/service.rs
+8
View File
@@ -510,6 +510,9 @@ async fn create_schedule(
if state.schedules.get(&req.id).await.is_some() { if state.schedules.get(&req.id).await.is_some() {
return Err((StatusCode::CONFLICT, format!("schedule '{}' already exists", req.id))); return Err((StatusCode::CONFLICT, format!("schedule '{}' already exists", req.id)));
} }
if let Err(e) = schedule::validate_trigger(&req.trigger) {
return Err((StatusCode::BAD_REQUEST, e));
}
let now = chrono::Utc::now(); let now = chrono::Utc::now();
let def = schedule::ScheduleDef { let def = schedule::ScheduleDef {
id: req.id, id: req.id,
@@ -564,6 +567,11 @@ async fn patch_schedule(
let Some(mut def) = state.schedules.get(&id).await else { let Some(mut def) = state.schedules.get(&id).await else {
return Err((StatusCode::NOT_FOUND, format!("schedule '{id}' not found"))); return Err((StatusCode::NOT_FOUND, format!("schedule '{id}' not found")));
}; };
if let Some(t) = &req.trigger {
if let Err(e) = schedule::validate_trigger(t) {
return Err((StatusCode::BAD_REQUEST, e));
}
}
if let Some(e) = req.enabled { def.enabled = e; } if let Some(e) = req.enabled { def.enabled = e; }
if let Some(t) = req.trigger { def.trigger = t; } if let Some(t) = req.trigger { def.trigger = t; }
match state.schedules.put(def).await { match state.schedules.put(def).await {
crates/storaged/src/append_log.rs
+97 -6
View File
@@ -18,7 +18,7 @@
/// to small JSONL events. /// to small JSONL events.
/// ///
/// Storage layout: /// Storage layout:
/// ``` /// ```text
/// {prefix}/ /// {prefix}/
/// batch_0001776319628000123.jsonl /// batch_0001776319628000123.jsonl
/// batch_0001776319745987654.jsonl /// batch_0001776319745987654.jsonl
@@ -221,15 +221,15 @@ pub struct CompactStats {
pub new_key: Option<String>, pub new_key: Option<String>,
} }
// Proactively flush on drop, best-effort. // Log unflushed-buffer size on drop. We can't `.await` from a sync `Drop`,
// We can't `.await` in Drop; we spawn the flush on the tokio runtime if one // so a real flush isn't possible here — callers are responsible for calling
// is available. If the runtime is already shutting down the flush is lost — // `.flush()` before dropping if durability matters. These journals are
// acceptable for these observability journals, which are hints not records. // observability hints; a few lost buffered events at shutdown are
// acceptable per ADR-018.
impl Drop for AppendLog { impl Drop for AppendLog {
fn drop(&mut self) { fn drop(&mut self) {
let buf_len = self.buffer.try_lock().map(|b| b.len()).unwrap_or(0); let buf_len = self.buffer.try_lock().map(|b| b.len()).unwrap_or(0);
if buf_len == 0 { return; } if buf_len == 0 { return; }
// Can't spawn from sync Drop on every runtime shape; log + move on.
tracing::debug!( tracing::debug!(
"append_log '{}' dropping with {} unflushed events", "append_log '{}' dropping with {} unflushed events",
self.prefix, buf_len, self.prefix, buf_len,
@@ -237,3 +237,94 @@ impl Drop for AppendLog {
} }
} }
#[cfg(test)]
mod tests {
use super::*;
use object_store::memory::InMemory;
fn mk(threshold: usize) -> AppendLog {
AppendLog::new(Arc::new(InMemory::new()), "prefix")
.with_flush_threshold(threshold)
}
#[tokio::test]
async fn append_stays_buffered_below_threshold() {
let log = mk(5);
log.append(b"one".to_vec()).await.unwrap();
log.append(b"two".to_vec()).await.unwrap();
assert_eq!(log.file_count().await.unwrap(), 0, "no files until threshold");
let all = log.read_all().await.unwrap();
assert_eq!(all, vec![b"one".to_vec(), b"two".to_vec()],
"read_all surfaces unflushed buffer");
}
#[tokio::test]
async fn append_auto_flushes_on_threshold() {
let log = mk(3);
for i in 0..3 {
log.append(format!("evt{i}").into_bytes()).await.unwrap();
}
assert_eq!(log.file_count().await.unwrap(), 1, "threshold triggered one flush");
// A fourth append stays buffered until the next threshold.
log.append(b"evt3".to_vec()).await.unwrap();
assert_eq!(log.file_count().await.unwrap(), 1, "below threshold again");
}
#[tokio::test]
async fn flush_empty_is_noop() {
let log = mk(32);
log.flush().await.unwrap();
log.flush().await.unwrap();
assert_eq!(log.file_count().await.unwrap(), 0);
}
#[tokio::test]
async fn read_all_orders_events_across_flushes() {
let log = mk(1); // flush-on-every-append
for i in 0..5 {
log.append(format!("e{i}").into_bytes()).await.unwrap();
// Spread writes out so timestamps sort strictly.
tokio::time::sleep(std::time::Duration::from_millis(2)).await;
}
let all = log.read_all().await.unwrap();
let strs: Vec<String> = all.into_iter()
.map(|v| String::from_utf8(v).unwrap())
.collect();
assert_eq!(strs, vec!["e0", "e1", "e2", "e3", "e4"],
"lex sort of batch keys == chronological event order");
}
#[tokio::test]
async fn compact_merges_multiple_files_into_one() {
let log = mk(1); // force file-per-append
for i in 0..4 {
log.append(format!("e{i}").into_bytes()).await.unwrap();
tokio::time::sleep(std::time::Duration::from_millis(2)).await;
}
assert_eq!(log.file_count().await.unwrap(), 4);
let stats = log.compact().await.unwrap();
assert_eq!(stats.merged_files, 4);
assert_eq!(stats.events, 4);
assert!(stats.new_key.is_some());
assert_eq!(log.file_count().await.unwrap(), 1, "originals deleted, 1 survivor");
let all = log.read_all().await.unwrap();
assert_eq!(all.len(), 4, "no events lost in compaction");
}
#[tokio::test]
async fn compact_with_single_file_is_noop() {
let log = mk(1);
log.append(b"only".to_vec()).await.unwrap();
assert_eq!(log.file_count().await.unwrap(), 1);
let stats = log.compact().await.unwrap();
assert_eq!(stats.merged_files, 1);
assert_eq!(stats.events, 0, "nothing to consolidate");
assert!(stats.new_key.is_none(), "no new file written");
assert_eq!(log.file_count().await.unwrap(), 1, "original untouched");
}
}
crates/storaged/src/error_journal.rs
+89
View File
@@ -198,3 +198,92 @@ impl ErrorJournal {
self.log.file_count().await self.log.file_count().await
} }
} }
#[cfg(test)]
mod tests {
use super::*;
use object_store::memory::InMemory;
fn mk() -> ErrorJournal {
ErrorJournal::new(Arc::new(InMemory::new()))
}
#[tokio::test]
async fn append_updates_ring_immediately() {
let j = mk();
j.append(BucketErrorEvent::new_read("tenant_a", "k1", "timeout")).await;
j.append(BucketErrorEvent::new_write("tenant_b", "k2", "denied")).await;
let recent = j.recent(10).await;
assert_eq!(recent.len(), 2);
assert_eq!(recent[0].target, "tenant_a");
assert_eq!(recent[1].target, "tenant_b");
}
#[tokio::test]
async fn load_recent_rehydrates_ring_from_disk() {
// Use a shared InMemory store so the second journal reads the first
// journal's writes — mirrors gateway restart semantics.
let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
{
let j1 = ErrorJournal {
log: Arc::new(AppendLog::new(store.clone(), JOURNAL_PREFIX).with_flush_threshold(1)),
ring: Arc::new(RwLock::new(VecDeque::with_capacity(RING_CAPACITY))),
};
j1.append(BucketErrorEvent::new_read("t", "k", "err")).await;
j1.flush().await.unwrap();
}
let j2 = ErrorJournal {
log: Arc::new(AppendLog::new(store.clone(), JOURNAL_PREFIX)),
ring: Arc::new(RwLock::new(VecDeque::with_capacity(RING_CAPACITY))),
};
assert_eq!(j2.recent(10).await.len(), 0, "fresh ring is empty");
let loaded = j2.load_recent().await.unwrap();
assert_eq!(loaded, 1);
assert_eq!(j2.recent(10).await[0].target, "t");
}
#[tokio::test]
async fn mark_rescued_updates_most_recent_match() {
let j = mk();
j.append(BucketErrorEvent::new_read("t1", "k", "err")).await;
j.append(BucketErrorEvent::new_read("t2", "k", "err")).await;
j.append(BucketErrorEvent::new_read("t1", "k", "err")).await;
j.mark_rescued_last("t1", "k").await;
let recent = j.recent(10).await;
assert!(!recent[0].rescued, "older t1 event not touched");
assert!(!recent[1].rescued, "t2 not touched");
assert!(recent[2].rescued, "most recent t1 was rescued");
}
#[tokio::test]
async fn health_flags_buckets_with_three_or_more_errors() {
let j = mk();
for _ in 0..3 { j.append(BucketErrorEvent::new_read("broken", "k", "e")).await; }
j.append(BucketErrorEvent::new_read("flaky", "k", "e")).await;
let h = j.health(60).await;
assert_eq!(h.total_errors, 4);
assert_eq!(h.per_bucket.get("broken"), Some(&3));
assert_eq!(h.per_bucket.get("flaky"), Some(&1));
assert!(h.unhealthy_buckets.contains(&"broken".to_string()));
assert!(!h.unhealthy_buckets.contains(&"flaky".to_string()));
}
#[tokio::test]
async fn filter_by_bucket_returns_chronological_order() {
let j = mk();
j.append(BucketErrorEvent::new_read("a", "k1", "e")).await;
j.append(BucketErrorEvent::new_read("b", "k2", "e")).await;
j.append(BucketErrorEvent::new_read("a", "k3", "e")).await;
let only_a = j.filter(Some("a"), None, 10).await;
assert_eq!(only_a.len(), 2);
assert_eq!(only_a[0].key, "k1", "oldest first");
assert_eq!(only_a[1].key, "k3");
}
}
crates/vectord-lance/src/lib.rs
+21 -1
View File
@@ -357,7 +357,25 @@ impl LanceVectorStore {
/// Search for top_k nearest neighbors of `query`. Uses the IVF_PQ /// Search for top_k nearest neighbors of `query`. Uses the IVF_PQ
/// index if one exists; otherwise does a full scan (slow but /// index if one exists; otherwise does a full scan (slow but
/// correct — useful during development before index build). /// correct — useful during development before index build).
pub async fn search(&self, query: &[f32], top_k: usize) -> Result<Vec<Hit>, String> { ///
/// `nprobes` tells Lance how many IVF partitions to probe per query.
/// Lance's built-in default is **1**, which caps recall well below
/// what the index is actually capable of. Passing `None` keeps that
/// default (only sensible when latency trumps recall); for real
/// workloads set `nprobes` to 5–10% of `num_partitions` (e.g. 20–30
/// on a 316-partition index).
///
/// `refine_factor` re-ranks the PQ-approximate top-k by computing
/// exact distances on `top_k * refine_factor` candidates, then
/// trimming to `top_k`. Cheap way to buy back recall lost to product
/// quantization. `None` skips the re-rank pass.
pub async fn search(
&self,
query: &[f32],
top_k: usize,
nprobes: Option<usize>,
refine_factor: Option<u32>,
) -> Result<Vec<Hit>, String> {
use lance::dataset::Dataset; use lance::dataset::Dataset;
let dataset = Dataset::open(&self.path).await.map_err(e)?; let dataset = Dataset::open(&self.path).await.map_err(e)?;
@@ -365,6 +383,8 @@ impl LanceVectorStore {
let mut scanner = dataset.scan(); let mut scanner = dataset.scan();
scanner.nearest("vector", &qarr, top_k as usize).map_err(e)?; scanner.nearest("vector", &qarr, top_k as usize).map_err(e)?;
if let Some(n) = nprobes { scanner.nprobes(n); }
if let Some(f) = refine_factor { scanner.refine(f); }
scanner.project(&["doc_id", "chunk_text"]).map_err(e)?; scanner.project(&["doc_id", "chunk_text"]).map_err(e)?;
let mut stream = scanner.try_into_stream().await.map_err(e)?; let mut stream = scanner.try_into_stream().await.map_err(e)?;
crates/vectord/Cargo.toml
+2
View File
@@ -9,6 +9,7 @@ storaged = { path = "../storaged" }
aibridge = { path = "../aibridge" } aibridge = { path = "../aibridge" }
catalogd = { path = "../catalogd" } catalogd = { path = "../catalogd" }
queryd = { path = "../queryd" } queryd = { path = "../queryd" }
futures = { workspace = true }
# ADR-019 firewall — vectord-lance owns its own Arrow 57 / Lance 4 deps. # ADR-019 firewall — vectord-lance owns its own Arrow 57 / Lance 4 deps.
# Public API uses only std types so no version conflict propagates here. # Public API uses only std types so no version conflict propagates here.
vectord-lance = { path = "../vectord-lance" } vectord-lance = { path = "../vectord-lance" }
@@ -24,3 +25,4 @@ arrow = { workspace = true }
chrono = { workspace = true } chrono = { workspace = true }
instant-distance = { workspace = true } instant-distance = { workspace = true }
uuid = { workspace = true } uuid = { workspace = true }
sha2 = { workspace = true }
crates/vectord/src/agent.rs
+111 -33
View File
@@ -204,6 +204,7 @@ pub struct AgentDeps {
pub embedding_cache: EmbeddingCache, pub embedding_cache: EmbeddingCache,
pub trial_journal: TrialJournal, pub trial_journal: TrialJournal,
pub promotion_registry: PromotionRegistry, pub promotion_registry: PromotionRegistry,
pub harness_store: crate::harness::HarnessStore,
} }
// -------- Spawn -------- // -------- Spawn --------
@@ -354,12 +355,11 @@ async fn run_one_cycle(
// Read history. // Read history.
let history = deps.trial_journal.list(&event.index_name).await let history = deps.trial_journal.list(&event.index_name).await
.map_err(|e| format!("read journal: {e}"))?; .map_err(|e| format!("read journal: {e}"))?;
if history.is_empty() {
return Err(format!( // Bootstrap mode: first ever visit to this index. Auto-generate a
"no trials yet for '{}' — seed with at least one POST /hnsw/trial first", // harness (if missing) and seed with the default config — the Phase 15
event.index_name, // known-good ec=80/es=30. Subsequent visits go through the proposer.
)); let is_bootstrap = history.is_empty();
}
// Current champion (if any) is the promoted config. // Current champion (if any) is the promoted config.
let champion = deps.promotion_registry.get_current(&event.index_name).await; let champion = deps.promotion_registry.get_current(&event.index_name).await;
@@ -367,24 +367,26 @@ async fn run_one_cycle(
history.iter().find(|t| t.id == p.trial_id).cloned() history.iter().find(|t| t.id == p.trial_id).cloned()
}); });
// Propose the next config. let (next_config, harness_name) = if is_bootstrap {
let Some(next_config) = propose_next_config(&history, champion_trial.as_ref()) else { let name = ensure_auto_harness(&event.index_name, deps).await?;
(HnswConfig::default(), name)
} else {
let Some(cfg) = propose_next_config(&history, champion_trial.as_ref()) else {
return Err("proposer returned None — search space exhausted".into()); return Err("proposer returned None — search space exhausted".into());
}; };
// Validate bounds defensively. // Validate bounds defensively.
if !(10..=400).contains(&next_config.ef_construction) { if !(10..=400).contains(&cfg.ef_construction) {
return Err(format!("proposed ef_construction={} out of bounds", next_config.ef_construction)); return Err(format!("proposed ef_construction={} out of bounds", cfg.ef_construction));
} }
if !(10..=200).contains(&next_config.ef_search) { if !(10..=200).contains(&cfg.ef_search) {
return Err(format!("proposed ef_search={} out of bounds", next_config.ef_search)); return Err(format!("proposed ef_search={} out of bounds", cfg.ef_search));
} }
// A future refinement: remember per-index "canonical harness" on
// Need a harness to measure. Use the most recent one in history. // the index metadata. For now: latest wins.
// (A future refinement: remember per-index "canonical harness" on let hname = history.last().unwrap().eval_set.clone();
// the index metadata. For now: latest wins.) (cfg, hname)
let harness_name = history.last().unwrap().eval_set.clone(); };
let mut harness_set = harness::EvalSet::load(&deps.store, &harness_name).await let mut harness_set = deps.harness_store.load_for_index(&event.index_name, &harness_name).await
.map_err(|e| format!("load harness '{harness_name}': {e}"))?; .map_err(|e| format!("load harness '{harness_name}': {e}"))?;
let embeddings = deps.embedding_cache.get_or_load(&event.index_name).await let embeddings = deps.embedding_cache.get_or_load(&event.index_name).await
@@ -393,7 +395,7 @@ async fn run_one_cycle(
if !harness_set.ground_truth_built { if !harness_set.ground_truth_built {
harness::compute_ground_truth(&mut harness_set, &embeddings, &deps.ai_client).await harness::compute_ground_truth(&mut harness_set, &embeddings, &deps.ai_client).await
.map_err(|e| format!("ground truth: {e}"))?; .map_err(|e| format!("ground truth: {e}"))?;
harness_set.save(&deps.store).await.ok(); deps.harness_store.save(&harness_set).await.ok();
} }
// Build + bench. // Build + bench.
@@ -428,11 +430,14 @@ async fn run_one_cycle(
let dims = embeddings.first().map(|e| e.vector.len()).unwrap_or(0); let dims = embeddings.first().map(|e| e.vector.len()).unwrap_or(0);
let memory_bytes = (embeddings.len() * dims * std::mem::size_of::<f32>() + embeddings.len() * 128) as u64; let memory_bytes = (embeddings.len() * dims * std::mem::size_of::<f32>() + embeddings.len() * 128) as u64;
let note = match &event.reason { let note = {
let base = match &event.reason {
TriggerReason::DatasetAppended { dataset } => format!("agent: dataset_appended({dataset})"), TriggerReason::DatasetAppended { dataset } => format!("agent: dataset_appended({dataset})"),
TriggerReason::Manual => "agent: manual".to_string(), TriggerReason::Manual => "agent: manual".to_string(),
TriggerReason::Periodic => "agent: periodic".to_string(), TriggerReason::Periodic => "agent: periodic".to_string(),
}; };
if is_bootstrap { format!("{base} bootstrap") } else { base }
};
let trial = Trial { let trial = Trial {
id: trial_id, id: trial_id,
@@ -597,19 +602,92 @@ pub fn propose_next_config(history: &[Trial], champion: Option<&Trial>) -> Optio
// -------- Helpers -------- // -------- Helpers --------
/// Find an index to poke on a periodic wake. Strategy: the one with the /// Minimum vectors for an index to be worth auto-tuning. Below this,
/// most recent trial. If nothing's been trialed yet, return None. /// HNSW's win over brute-force is too small to matter and trial budget
/// is better spent on bigger indexes.
const AUTOTUNE_MIN_VECTORS: usize = 1_000;
/// Find an index to poke on a periodic wake. Strategy: least-recently-tuned
/// wins — pick the index (from `IndexRegistry`, not just promoted ones)
/// whose most-recent trial is oldest, so trial budget spreads across the
/// whole portfolio. Never-trialed indexes sort first (None < Some), which
/// is exactly what we want: bootstrap them on their first visit.
///
/// Why not "most recently promoted" (the original strategy): a converged
/// index like `threat_intel_v1` sits at recall=1.0 on a minimal graph and
/// can't be improved — but it was always the freshest promotion, so the
/// agent burned every trial on it while 40+ other indexes got zero
/// attention. Rotating by last-trial-time fixes that without any explicit
/// convergence detection.
///
/// Why IndexRegistry instead of PromotionRegistry: on a fresh system only
/// a handful of indexes have ever been promoted, so promotion-based
/// picking starves new ones. IndexRegistry lists every live index the
/// platform knows about; filtering by `AUTOTUNE_MIN_VECTORS` keeps the
/// proposer off indexes where the graph config doesn't matter.
///
/// Cost: N extra journal reads per periodic tick, where N = eligible
/// indexes. Fine at dozens; if the portfolio grows into thousands, cache
/// `last_trial_at` on the IndexMeta.
async fn pick_periodic_target(deps: &AgentDeps) -> Option<String> { async fn pick_periodic_target(deps: &AgentDeps) -> Option<String> {
// `/agent` runs against any index that has a trial journal. We don't let candidates: Vec<String> = deps
// have a "list all journals" helper, so we derive candidates from the .index_registry
// promotion registry (indexes with a human ever promoting are live). .list(None, None)
let promos = deps.promotion_registry.list_all().await.ok()?; .await
// Prefer the one most recently promoted — it's the one a human cares .into_iter()
// about right now. .filter(|m| m.chunk_count >= AUTOTUNE_MIN_VECTORS)
promos.into_iter() .map(|m| m.index_name)
.filter_map(|f| f.current.map(|c| (f.index_name, c.promoted_at))) .collect();
.max_by_key(|(_, at)| *at) if candidates.is_empty() { return None; }
.map(|(name, _)| name)
let mut scored: Vec<(String, Option<DateTime<Utc>>)> = Vec::with_capacity(candidates.len());
for name in candidates {
let last_trial_at = deps
.trial_journal
.list(&name)
.await
.ok()
.and_then(|trials| trials.into_iter().map(|t| t.created_at).max());
scored.push((name, last_trial_at));
}
scored.into_iter().min_by_key(|(_, last)| *last).map(|(n, _)| n)
}
/// On the first visit to an index, load or synthesize an eval harness so
/// bootstrap trials have something to measure recall against. Returns the
/// harness name. Synthetic harnesses sample 20 chunks and use their first
/// 200 chars as self-queries — the ground truth is computed by brute-force
/// cosine, so recall numbers are real (a config that finds the chunk in
/// top-k counts as a hit).
async fn ensure_auto_harness(index_name: &str, deps: &AgentDeps) -> Result<String, String> {
let name = format!("{index_name}_auto");
if deps.harness_store.load_for_index(index_name, &name).await.is_ok() {
return Ok(name);
}
let embeddings = deps
.embedding_cache
.get_or_load(index_name)
.await
.map_err(|e| format!("load embeddings for auto-harness: {e}"))?;
if embeddings.is_empty() {
return Err(format!("index '{index_name}' has no embeddings — cannot bootstrap"));
}
let mut eval = harness::synthetic_from_chunks(&name, index_name, &embeddings, 20, 10);
harness::compute_ground_truth(&mut eval, &embeddings, &deps.ai_client)
.await
.map_err(|e| format!("auto-harness ground truth: {e}"))?;
deps.harness_store.save(&eval).await
.map_err(|e| format!("save auto-harness: {e}"))?;
tracing::info!(
"agent: bootstrapped harness '{}' for index '{}' (20 self-queries, k=10)",
name, index_name
);
Ok(name)
} }
async fn over_rate_limit(inner: &Arc<AgentInner>, cap: u32) -> bool { async fn over_rate_limit(inner: &Arc<AgentInner>, cap: u32) -> bool {
crates/vectord/src/autotune.rs
+9 -5
View File
@@ -28,7 +28,7 @@ use aibridge::client::AiClient;
use catalogd::registry::Registry as CatalogRegistry; use catalogd::registry::Registry as CatalogRegistry;
use crate::embedding_cache::EmbeddingCache; use crate::embedding_cache::EmbeddingCache;
use crate::harness; use crate::harness::{self, HarnessStore};
use crate::hnsw::HnswStore; use crate::hnsw::HnswStore;
use crate::index_registry::IndexRegistry; use crate::index_registry::IndexRegistry;
use crate::jobs::JobTracker; use crate::jobs::JobTracker;
@@ -128,7 +128,7 @@ fn pick_winner(trials: &[Trial], min_recall: f32) -> Option<&Trial> {
#[allow(clippy::too_many_arguments)] #[allow(clippy::too_many_arguments)]
pub async fn run_autotune( pub async fn run_autotune(
req: AutotuneRequest, req: AutotuneRequest,
store: &Arc<dyn ObjectStore>, _store: &Arc<dyn ObjectStore>,
catalog: &CatalogRegistry, catalog: &CatalogRegistry,
ai_client: &AiClient, ai_client: &AiClient,
embedding_cache: &EmbeddingCache, embedding_cache: &EmbeddingCache,
@@ -136,6 +136,7 @@ pub async fn run_autotune(
index_registry: &IndexRegistry, index_registry: &IndexRegistry,
trial_journal: &TrialJournal, trial_journal: &TrialJournal,
promotion_registry: &PromotionRegistry, promotion_registry: &PromotionRegistry,
harness_store: &HarnessStore,
_job_tracker: &JobTracker, _job_tracker: &JobTracker,
) -> Result<AutotuneResult, String> { ) -> Result<AutotuneResult, String> {
let t0 = std::time::Instant::now(); let t0 = std::time::Instant::now();
@@ -146,8 +147,11 @@ pub async fn run_autotune(
} }
let _ = catalog; // reserved for future audit emission let _ = catalog; // reserved for future audit emission
// Load the harness once, compute ground truth once. // Load the harness once, compute ground truth once. Harness resolves
let mut harness_set = harness::EvalSet::load(store, &req.harness) // to the index's bucket via HarnessStore, with primary as fallback for
// pre-federation evals.
let mut harness_set = harness_store
.load_for_index(&req.index_name, &req.harness)
.await .await
.map_err(|e| format!("load harness: {e}"))?; .map_err(|e| format!("load harness: {e}"))?;
@@ -160,7 +164,7 @@ pub async fn run_autotune(
harness::compute_ground_truth(&mut harness_set, &embeddings, ai_client) harness::compute_ground_truth(&mut harness_set, &embeddings, ai_client)
.await .await
.map_err(|e| format!("ground truth: {e}"))?; .map_err(|e| format!("ground truth: {e}"))?;
harness_set.save(store).await.ok(); harness_store.save(&harness_set).await.ok();
} }
let (grid, rejected) = sanitize_grid(req.grid.clone().unwrap_or_else(default_grid)); let (grid, rejected) = sanitize_grid(req.grid.clone().unwrap_or_else(default_grid));
crates/vectord/src/harness.rs
+176 -5
View File
@@ -10,11 +10,14 @@
use chrono::{DateTime, Utc}; use chrono::{DateTime, Utc};
use object_store::ObjectStore; use object_store::ObjectStore;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use std::collections::HashSet;
use std::sync::Arc; use std::sync::Arc;
use aibridge::client::{AiClient, EmbedRequest}; use aibridge::client::{AiClient, EmbedRequest};
use storaged::ops; use storaged::ops;
use storaged::registry::BucketRegistry;
use crate::index_registry::IndexRegistry;
use crate::store::StoredEmbedding; use crate::store::StoredEmbedding;
/// A single eval query with optional pre-computed ground truth. /// A single eval query with optional pre-computed ground truth.
@@ -85,6 +88,108 @@ impl EvalSet {
} }
} }
/// Federation-aware wrapper around EvalSet persistence. Mirrors the
/// `TrialJournal` / `PromotionRegistry` pattern: harness files colocate
/// with their index's bucket (looked up via `IndexMeta.bucket`), falling
/// back to `primary` for indexes the registry has never seen. Legacy
/// harnesses predating federation remain discoverable — lookups transparently
/// try the resolved bucket first, then `primary` as a fallback. Cross-bucket
/// listing dedupes across every registered bucket so `GET /hnsw/evals`
/// returns a complete picture.
#[derive(Clone)]
pub struct HarnessStore {
buckets: Arc<BucketRegistry>,
index_registry: IndexRegistry,
}
impl HarnessStore {
pub fn new(buckets: Arc<BucketRegistry>, index_registry: IndexRegistry) -> Self {
Self { buckets, index_registry }
}
/// Resolve which bucket holds this index's eval artifacts. Indexes the
/// registry has never heard of fall through to `primary`.
async fn bucket_for_index(&self, index_name: &str) -> String {
self.index_registry
.get(index_name)
.await
.map(|m| m.bucket)
.unwrap_or_else(|| "primary".to_string())
}
/// Save to the bucket that owns `eval.index_name`. Writes under the
/// standard `_hnsw_evals/{name}.json` prefix of the resolved bucket.
pub async fn save(&self, eval: &EvalSet) -> Result<(), String> {
let bucket = self.bucket_for_index(&eval.index_name).await;
let store = self.buckets.get(&bucket)?;
eval.save(&store).await
}
/// Load a harness by name, given the index it belongs to. Tries the
/// index's bucket first; if the file is absent AND the resolved bucket
/// isn't `primary`, falls through to `primary` so pre-federation evals
/// remain reachable without migration.
pub async fn load_for_index(
&self,
index_name: &str,
harness_name: &str,
) -> Result<EvalSet, String> {
let bucket = self.bucket_for_index(index_name).await;
let primary_store = self.buckets.get("primary")?;
let store = self.buckets.get(&bucket)?;
match EvalSet::load(&store, harness_name).await {
Ok(e) => Ok(e),
Err(e) if bucket != "primary" => EvalSet::load(&primary_store, harness_name)
.await
.map_err(|primary_err| format!("{bucket}: {e}; primary fallback: {primary_err}")),
Err(e) => Err(e),
}
}
/// Find a harness by name without knowing which index it belongs to —
/// used by `GET /hnsw/evals/{name}`. Scans every registered bucket;
/// first hit wins. Primary is searched first so pre-federation evals
/// with the same name as a federated one resolve deterministically.
pub async fn get_any(&self, harness_name: &str) -> Result<EvalSet, String> {
let bucket_infos = self.buckets.list().await;
let mut ordered: Vec<String> = bucket_infos.iter().map(|b| b.name.clone()).collect();
ordered.sort_by_key(|n| if n == "primary" { 0 } else { 1 });
let mut last_err = None;
for b in ordered {
let store = match self.buckets.get(&b) {
Ok(s) => s,
Err(e) => { last_err = Some(e); continue; }
};
match EvalSet::load(&store, harness_name).await {
Ok(e) => return Ok(e),
Err(e) => { last_err = Some(e); }
}
}
Err(last_err.unwrap_or_else(|| format!("no buckets registered for eval '{harness_name}'")))
}
/// Union of every harness name across every registered bucket.
/// Duplicates (same name in multiple buckets — pathological but
/// possible after a manual migration) are collapsed.
pub async fn list_all(&self) -> Vec<String> {
let mut all: HashSet<String> = HashSet::new();
for b in self.buckets.list().await {
let store = match self.buckets.get(&b.name) {
Ok(s) => s,
Err(_) => continue,
};
if let Ok(names) = EvalSet::list(&store).await {
all.extend(names);
}
}
let mut out: Vec<String> = all.into_iter().collect();
out.sort();
out
}
}
/// Cosine similarity for two same-length f32 slices. /// Cosine similarity for two same-length f32 slices.
fn cosine(a: &[f32], b: &[f32]) -> f32 { fn cosine(a: &[f32], b: &[f32]) -> f32 {
let mut dot = 0.0f32; let mut dot = 0.0f32;
@@ -127,13 +232,15 @@ pub fn recall_at_k(predicted: &[String], ground_truth: &[String], k: usize) -> f
if ground_truth.is_empty() || k == 0 { if ground_truth.is_empty() || k == 0 {
return 0.0; return 0.0;
} }
// Set-intersection recall@k. Previous implementation counted duplicates
// in `predicted` (a corpus with repeated chunks — e.g. cached LLM
// responses — returns the same doc_id multiple times via HNSW), which
// inflated recall above 1.0 and poisoned promotion decisions.
let gt_set: std::collections::HashSet<&String> = let gt_set: std::collections::HashSet<&String> =
ground_truth.iter().take(k).collect(); ground_truth.iter().take(k).collect();
let hits = predicted let pred_set: std::collections::HashSet<&String> =
.iter() predicted.iter().take(k).collect();
.take(k) let hits = pred_set.intersection(&gt_set).count();
.filter(|d| gt_set.contains(d))
.count();
hits as f32 / gt_set.len() as f32 hits as f32 / gt_set.len() as f32
} }
@@ -214,3 +321,67 @@ pub fn synthetic_from_chunks(
ground_truth_built: false, ground_truth_built: false,
} }
} }
#[cfg(test)]
mod tests {
use super::*;
fn s(v: &[&str]) -> Vec<String> { v.iter().map(|x| x.to_string()).collect() }
#[test]
fn recall_empty_ground_truth_is_zero() {
assert_eq!(recall_at_k(&s(&["a", "b"]), &[], 10), 0.0);
}
#[test]
fn recall_k_zero_is_zero() {
assert_eq!(recall_at_k(&s(&["a"]), &s(&["a"]), 0), 0.0);
}
#[test]
fn recall_perfect_match_equals_one() {
let pred = s(&["a", "b", "c"]);
let gt = s(&["a", "b", "c"]);
assert!((recall_at_k(&pred, &gt, 3) - 1.0).abs() < 1e-6);
}
#[test]
fn recall_half_match() {
let pred = s(&["a", "b", "x", "y"]);
let gt = s(&["a", "b", "c", "d"]);
assert!((recall_at_k(&pred, &gt, 4) - 0.5).abs() < 1e-6);
}
#[test]
fn recall_duplicates_in_predicted_do_not_inflate() {
// Regression guard: the previous implementation counted each
// duplicate in `predicted` separately, which could push recall
// above 1.0 on corpora with repeated chunks (cached responses etc).
// Set-intersection semantics keep it bounded in [0, 1].
let pred = s(&["a", "a", "a", "a"]);
let gt = s(&["a", "b", "c", "d"]);
let r = recall_at_k(&pred, &gt, 4);
assert!(r <= 1.0, "recall {r} must not exceed 1.0");
// One unique match out of four in gt = 0.25.
assert!((r - 0.25).abs() < 1e-6);
}
#[test]
fn recall_duplicates_in_ground_truth_handled() {
// gt with dupes reduces effective |gt|; matching one hits all.
let pred = s(&["x"]);
let gt = s(&["x", "x", "x"]);
let r = recall_at_k(&pred, &gt, 3);
assert!(r <= 1.0);
assert!((r - 1.0).abs() < 1e-6); // |pred ∩ gt| / |gt_set| = 1/1
}
#[test]
fn recall_respects_k_bound() {
// k=2 means only the first 2 of pred and gt count toward the set.
let pred = s(&["a", "b", "c", "d"]);
let gt = s(&["a", "b", "c", "d"]);
let r = recall_at_k(&pred, &gt, 2);
assert!((r - 1.0).abs() < 1e-6);
}
}
crates/vectord/src/playbook_memory.rs
+1016 -15
View File
File diff suppressed because it is too large Load Diff
crates/vectord/src/rag.rs
+26 -10
View File
@@ -10,6 +10,7 @@ use object_store::ObjectStore;
use std::sync::Arc; use std::sync::Arc;
use aibridge::client::{AiClient, EmbedRequest, GenerateRequest}; use aibridge::client::{AiClient, EmbedRequest, GenerateRequest};
use aibridge::continuation::{generate_continuable, ContinuableOpts, ResponseShape};
use crate::search::{self, SearchResult}; use crate::search::{self, SearchResult};
use crate::store; use crate::store;
@@ -42,6 +43,10 @@ async fn rerank(
system: None, system: None,
temperature: Some(0.0), temperature: Some(0.0),
max_tokens: Some(50), max_tokens: Some(50),
// Reranker returns a comma-separated int list — pure structured
// output, zero benefit from hidden reasoning. Opt out to avoid
// the empty-response failure mode Phase 21 catalogued.
think: Some(false),
}).await; }).await;
match resp { match resp {
@@ -150,18 +155,29 @@ pub async fn query(
Answer:" Answer:"
); );
let gen_resp = ai_client.generate(GenerateRequest { // Route the answer call through Phase 21's generate_continuable so
prompt, // a thinking-model empty-response or a mid-JSON truncation self-
model: None, // repairs instead of silently returning half an answer. Shape is
system: None, // Text (the answer is prose, not JSON), think is Some(false) to
temperature: Some(0.2), // opt out of hidden reasoning on the hot path. This is the first
max_tokens: Some(512), // production caller of the Phase 21 primitives — see audit finding
}).await?; // "Phase 21 Rust primitives are wired but not CALLED by any
// production surface" from 2026-04-21.
let mut cont_opts = ContinuableOpts::new("qwen2.5:latest");
cont_opts.max_tokens = Some(512);
cont_opts.temperature = Some(0.2);
cont_opts.shape = ResponseShape::Text;
cont_opts.think = Some(false);
let outcome = generate_continuable(ai_client, &prompt, &cont_opts).await?;
Ok(RagResponse { Ok(RagResponse {
answer: gen_resp.text.trim().to_string(), answer: outcome.text.trim().to_string(),
model: gen_resp.model, // generate_continuable doesn't surface the model name (sidecar
// echoes whatever Ollama loaded). Use the configured tier model
// for now; if RAG needs to report the actual resolved model,
// the runner can add a post-call ps probe later.
model: "qwen2.5:latest".to_string(),
sources: results, sources: results,
tokens_generated: gen_resp.tokens_generated, tokens_generated: None,
}) })
} }
crates/vectord/src/service.rs
+464 -11
View File
@@ -13,6 +13,7 @@ use aibridge::client::{AiClient, EmbedRequest, GenerateRequest};
use catalogd::registry::Registry as CatalogRegistry; use catalogd::registry::Registry as CatalogRegistry;
use storaged::registry::BucketRegistry; use storaged::registry::BucketRegistry;
use crate::{agent, autotune, chunker, embedding_cache, harness, hnsw, index_registry, jobs, lance_backend, playbook_memory, promotion, rag, refresh, search, store, supervisor, trial}; use crate::{agent, autotune, chunker, embedding_cache, harness, hnsw, index_registry, jobs, lance_backend, playbook_memory, promotion, rag, refresh, search, store, supervisor, trial};
use tokio::sync::Semaphore;
#[derive(Clone)] #[derive(Clone)]
pub struct VectorState { pub struct VectorState {
@@ -53,6 +54,9 @@ pub struct VectorState {
/// and, when `use_playbook_memory` is set on /vectors/hybrid, boosts /// and, when `use_playbook_memory` is set on /vectors/hybrid, boosts
/// workers that were actually filled in semantically-similar past ops. /// workers that were actually filled in semantically-similar past ops.
pub playbook_memory: playbook_memory::PlaybookMemory, pub playbook_memory: playbook_memory::PlaybookMemory,
/// Serializes embed calls from seed_playbook_memory to avoid
/// concurrent socket collisions with the Python sidecar.
pub embed_semaphore: Arc<Semaphore>,
} }
/// What the active-profile singleton records. Narrow — we don't need the /// What the active-profile singleton records. Narrow — we don't need the
@@ -124,6 +128,11 @@ pub fn router(state: VectorState) -> Router {
.route("/playbook_memory/seed", post(seed_playbook_memory)) .route("/playbook_memory/seed", post(seed_playbook_memory))
.route("/playbook_memory/persist_sql", post(persist_playbook_memory_sql)) .route("/playbook_memory/persist_sql", post(persist_playbook_memory_sql))
.route("/playbook_memory/patterns", post(discover_playbook_patterns)) .route("/playbook_memory/patterns", post(discover_playbook_patterns))
.route("/playbook_memory/mark_failed", post(mark_playbook_failed))
.route("/playbook_memory/retire", post(retire_playbook_memory))
.route("/playbook_memory/revise", post(revise_playbook_memory))
.route("/playbook_memory/history/{id}", get(playbook_memory_history))
.route("/playbook_memory/status", get(playbook_memory_status))
.with_state(state) .with_state(state)
} }
@@ -802,22 +811,65 @@ async fn hybrid_search(
// set. Additive boost on the existing vector score, then re-sort. // set. Additive boost on the existing vector score, then re-sort.
if req.use_playbook_memory { if req.use_playbook_memory {
let boost_k = req.playbook_memory_k.unwrap_or(playbook_memory::DEFAULT_TOP_K_PLAYBOOKS); let boost_k = req.playbook_memory_k.unwrap_or(playbook_memory::DEFAULT_TOP_K_PLAYBOOKS);
// Extract target (city, state, role) from the SQL filter so
// compute_boost_for can skip playbooks from other cities AND
// prioritize exact role matches via the multi-strategy path.
// The executor's filter shape is stable:
// `... role = 'Welder' AND city = 'Toledo' AND state = 'OH' ...`.
// Case-insensitive match, tolerant of single quotes.
let target_geo = req.sql_filter.as_deref().and_then(extract_target_geo);
let target_role = req.sql_filter.as_deref().and_then(extract_target_role);
// We embedded the question as `qv` above — reuse it for the // We embedded the question as `qv` above — reuse it for the
// playbook similarity lookup so we don't double-pay Ollama. // playbook similarity lookup so we don't double-pay Ollama.
let boosts = state.playbook_memory.compute_boost_for(&qv, boost_k, 0.5).await; let boosts = state.playbook_memory
.compute_boost_for_filtered_with_role(
&qv,
boost_k,
0.5,
target_geo.as_ref().map(|(c, s)| (c.as_str(), s.as_str())),
target_role.as_deref(),
)
.await;
// Diagnostics for Phase 19 boost pipeline. Logged so item 3
// investigation has ground truth:
// - boosts.len(): how many (city,state,name) keys surfaced for
// this query (0 = playbook_memory found nothing semantically
// similar to the question).
// - parsed: how many candidate chunks parsed cleanly into
// (name,city,state) via parse_worker_chunk.
// - matched: how many parsed keys matched an entry in boosts.
// 2026-04-21 — 20-scenario batch showed 34/40 ok combos never
// got a citation. These counters pin whether the gap is on the
// SIMILARITY side (boosts empty) or the MATCH side (parsed vs
// boosted keys mismatch — e.g. name format drift).
let mut parsed_count = 0usize;
let mut matched_count = 0usize;
for src in sources.iter_mut() { for src in sources.iter_mut() {
// Parse "{Name} — {Role} in {City}, {State}. …" chunk. Being // Parse "{Name} — {Role} in {City}, {State}. …" chunk. Being
// defensive: chunks from other datasets may not follow this // defensive: chunks from other datasets may not follow this
// exact shape, so absent fields just skip the boost. // exact shape, so absent fields just skip the boost.
if let Some((name, city, state)) = parse_worker_chunk(&src.chunk_text) { if let Some((name, city, state)) = parse_worker_chunk(&src.chunk_text) {
parsed_count += 1;
let key = (city, state, name); let key = (city, state, name);
if let Some(entry) = boosts.get(&key) { if let Some(entry) = boosts.get(&key) {
src.score += entry.boost; src.score += entry.boost;
src.playbook_boost = entry.boost; src.playbook_boost = entry.boost;
src.playbook_citations = entry.citations.clone(); src.playbook_citations = entry.citations.clone();
matched_count += 1;
} }
} }
} }
tracing::info!(
"playbook_boost: boosts={} sources={} parsed={} matched={} target_geo={:?} target_role={:?} (query='{}')",
boosts.len(),
sources.len(),
parsed_count,
matched_count,
target_geo,
target_role,
req.question.chars().take(60).collect::<String>(),
);
// Re-rank: boosted scores can flip ordering. // Re-rank: boosted scores can flip ordering.
sources.sort_by(|a, b| b.score.partial_cmp(&a.score).unwrap_or(std::cmp::Ordering::Equal)); sources.sort_by(|a, b| b.score.partial_cmp(&a.score).unwrap_or(std::cmp::Ordering::Equal));
// Finally trim to the caller's requested top_k — we pulled fetch_k // Finally trim to the caller's requested top_k — we pulled fetch_k
@@ -845,6 +897,9 @@ async fn hybrid_search(
system: None, system: None,
temperature: Some(0.2), temperature: Some(0.2),
max_tokens: Some(512), max_tokens: Some(512),
// Hybrid's answer step — prose output over retrieved records,
// no reasoning needed on the hot path.
think: Some(false),
}).await; }).await;
gen_resp.ok().map(|r| r.text.trim().to_string()) gen_resp.ok().map(|r| r.text.trim().to_string())
@@ -1534,7 +1589,7 @@ async fn activate_profile(
}); });
} }
Ok(Json(ActivateReport { let report = ActivateReport {
profile_id: profile.id, profile_id: profile.id,
ollama_name: profile.ollama_name, ollama_name: profile.ollama_name,
indexes_warmed: warmed, indexes_warmed: warmed,
@@ -1543,7 +1598,9 @@ async fn activate_profile(
duration_secs: t0.elapsed().as_secs_f32(), duration_secs: t0.elapsed().as_secs_f32(),
model_preloaded, model_preloaded,
previous_profile: previous_slot.map(|s| s.profile_id), previous_profile: previous_slot.map(|s| s.profile_id),
})) };
Ok(Json(report))
} }
/// Unload this profile's model and clear the active slot. No-op if the /// Unload this profile's model and clear the active slot. No-op if the
@@ -2058,6 +2115,56 @@ struct LanceRecallQuery {
/// "{Name} — {Role} in {City}, {State}. Skills: …". /// "{Name} — {Role} in {City}, {State}. Skills: …".
/// Returns None if the chunk doesn't match the shape; callers simply /// Returns None if the chunk doesn't match the shape; callers simply
/// skip the boost for that hit. /// skip the boost for that hit.
/// Extract role from an SQL filter matching `role = 'Welder'` style.
/// Case-insensitive on the column name. Quoted value; quotes not
/// included in returned string.
fn extract_target_role(sql_filter: &str) -> Option<String> {
grab_eq_value(sql_filter, "role")
}
/// Shared equality-value extractor for (city, state, role) lookups.
fn grab_eq_value(src: &str, col: &str) -> Option<String> {
let lower = src.to_ascii_lowercase();
let col_lower = col.to_ascii_lowercase();
let mut search_from = 0usize;
while let Some(off) = lower[search_from..].find(&col_lower) {
let pos = search_from + off;
let prior_ok = pos == 0
|| !lower.as_bytes()[pos - 1].is_ascii_alphanumeric()
&& lower.as_bytes()[pos - 1] != b'_';
let after = pos + col_lower.len();
if !prior_ok || after >= src.len() {
search_from = pos + col_lower.len();
continue;
}
let mut i = after;
while i < src.len() && src.as_bytes()[i] == b' ' { i += 1; }
if i >= src.len() || src.as_bytes()[i] != b'=' { search_from = pos + col_lower.len(); continue; }
i += 1;
while i < src.len() && src.as_bytes()[i] == b' ' { i += 1; }
if i >= src.len() || src.as_bytes()[i] != b'\'' { search_from = pos + col_lower.len(); continue; }
i += 1;
let start = i;
while i < src.len() && src.as_bytes()[i] != b'\'' { i += 1; }
if i > start {
return Some(src[start..i].to_string());
}
search_from = pos + col_lower.len();
}
None
}
/// Pull (city, state) out of a SQL filter that uses
/// `city = 'Toledo' AND state = 'OH'` style equality. Returns None if
/// either is missing — the caller keeps the original global boost map
/// behavior (no geo narrowing). Case-insensitive on the column name
/// so `CITY=` or `City =` also work.
fn extract_target_geo(sql_filter: &str) -> Option<(String, String)> {
let city = grab_eq_value(sql_filter, "city")?;
let state = grab_eq_value(sql_filter, "state")?;
Some((city, state))
}
fn parse_worker_chunk(chunk: &str) -> Option<(String, String, String)> { fn parse_worker_chunk(chunk: &str) -> Option<(String, String, String)> {
// "Name — Role in City, ST. …" → split on "—" then " in " then "," // "Name — Role in City, ST. …" → split on "—" then " in " then ","
let (name_part, rest) = chunk.split_once('—')?; let (name_part, rest) = chunk.split_once('—')?;
@@ -2091,6 +2198,25 @@ struct SeedPlaybookRequest {
/// seeding is a bootstrap/demo tool, not a rebuild substitute. /// seeding is a bootstrap/demo tool, not a rebuild substitute.
#[serde(default = "default_true")] #[serde(default = "default_true")]
append: bool, append: bool,
/// Phase 25 — optional schema_fingerprint captured at seed time.
/// When the underlying dataset's schema changes, any entry whose
/// fingerprint doesn't match the new one is auto-retired via
/// retire_on_schema_drift. Caller-provided so the producer (the
/// scenario driver, the orchestrator) can pass the live fingerprint
/// without the gateway needing a second catalogd round trip.
#[serde(default)]
schema_fingerprint: Option<String>,
/// Phase 25 — optional hard expiry. RFC3339 timestamp. After this
/// moment the entry is skipped during boost computation (not
/// retired, just inactive). Useful for seasonal/temp contracts.
#[serde(default)]
valid_until: Option<String>,
/// Phase 45 — optional external doc references captured at seal
/// time. Each entry names a tool + version_seen; context7-driven
/// drift check compares against current versions later. None or
/// empty = no drift signal (never flagged).
#[serde(default)]
doc_refs: Option<Vec<playbook_memory::DocRef>>,
} }
/// Bootstrap / test-only: inject a playbook entry directly into /// Bootstrap / test-only: inject a playbook entry directly into
@@ -2105,16 +2231,19 @@ async fn seed_playbook_memory(
State(state): State<VectorState>, State(state): State<VectorState>,
Json(req): Json<SeedPlaybookRequest>, Json(req): Json<SeedPlaybookRequest>,
) -> impl IntoResponse { ) -> impl IntoResponse {
// Serialize embed calls to avoid concurrent socket collisions.
let _permit = state.embed_semaphore.acquire().await.map_err(|e|
(StatusCode::INTERNAL_SERVER_ERROR, format!("semaphore error: {e}")))?;
// Embed the entry through the same text shape `rebuild` uses so // Embed the entry through the same text shape `rebuild` uses so
// similarity math is comparable across seed + real entries. // similarity math is comparable across seed + real entries.
let tmp_entry = playbook_memory::PlaybookEntry { let tmp_entry = playbook_memory::PlaybookEntry {
playbook_id: String::new(),
operation: req.operation.clone(), operation: req.operation.clone(),
approach: req.approach.clone(), approach: req.approach.clone(),
context: req.context.clone(), context: req.context.clone(),
timestamp: chrono::Utc::now().to_rfc3339(), timestamp: chrono::Utc::now().to_rfc3339(),
endorsed_names: req.endorsed_names.clone(), endorsed_names: req.endorsed_names.clone(),
city: None, state: None, embedding: None, ..Default::default()
}; };
let text = format!( let text = format!(
"{} | {} | {} | fills: {}", "{} | {} | {} | fills: {}",
@@ -2166,18 +2295,43 @@ async fn seed_playbook_memory(
endorsed_names: req.endorsed_names, endorsed_names: req.endorsed_names,
city, state: state_, city, state: state_,
embedding: Some(emb), embedding: Some(emb),
// Phase 25 — seed request may carry a fingerprint; if not, we
// default to None and the entry degrades to "no expiry signal"
// (never auto-retired on drift, but manual retirement still
// works). valid_until + retired_at start None.
schema_fingerprint: req.schema_fingerprint.clone(),
valid_until: req.valid_until.clone(),
// Phase 45 — seed request may also carry doc_refs; defaults
// empty so pre-Phase-45 callers still work and the entry
// degrades to "no drift signal" (never flagged).
doc_refs: req.doc_refs.clone().unwrap_or_default(),
..Default::default()
}; };
let mut current = state.playbook_memory.snapshot().await; // Phase 26 — when append=true (default), route through upsert so
// same-day re-seeds of the same operation merge instead of
// appending duplicates. When append=false, retain the old
// replace-all semantics for callers that want a hard reset.
if req.append { if req.append {
current.push(new_entry); match state.playbook_memory.upsert_entry(new_entry).await {
} else { Ok(outcome) => {
current = vec![new_entry]; let entries_after = state.playbook_memory.entry_count().await;
Ok(Json(serde_json::json!({
"outcome": outcome,
"entries_after": entries_after,
})))
} }
if let Err(e) = state.playbook_memory.set_entries(current).await { Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, format!("upsert: {e}"))),
}
} else {
if let Err(e) = state.playbook_memory.set_entries(vec![new_entry]).await {
return Err((StatusCode::INTERNAL_SERVER_ERROR, format!("persist: {e}"))); return Err((StatusCode::INTERNAL_SERVER_ERROR, format!("persist: {e}")));
} }
Ok(Json(serde_json::json!({ "playbook_id": pid, "entries_after": state.playbook_memory.entry_count().await }))) Ok(Json(serde_json::json!({
"outcome": { "mode": "replaced", "playbook_id": pid },
"entries_after": state.playbook_memory.entry_count().await,
})))
}
} }
async fn rebuild_playbook_memory( async fn rebuild_playbook_memory(
@@ -2242,6 +2396,47 @@ async fn discover_playbook_patterns(
} }
} }
#[derive(Deserialize)]
struct MarkFailedRequest {
/// Operation text, same shape as seed: "fill: Role xN in City, ST"
operation: String,
/// Names of workers who didn't deliver on the fill.
failed_names: Vec<String>,
/// Short reason (no-show, fired, unreliable). Stored verbatim.
#[serde(default)]
reason: String,
}
async fn mark_playbook_failed(
State(state): State<VectorState>,
Json(req): Json<MarkFailedRequest>,
) -> impl IntoResponse {
// Parse city + state from the operation — mirrors seed's parser.
let after_in = req.operation.split(" in ").nth(1).unwrap_or("");
let mut parts = after_in.splitn(2, ',');
let city = parts.next().map(|s| s.trim().to_string()).filter(|s| !s.is_empty());
let state_ = parts.next().map(|s|
s.trim().chars().take_while(|c| c.is_ascii_alphabetic()).collect::<String>()
).filter(|s| !s.is_empty());
let (Some(city), Some(state_code)) = (city, state_) else {
return Err((StatusCode::BAD_REQUEST,
"operation must match 'fill: Role xN in City, ST' shape".into()));
};
let ts = chrono::Utc::now().to_rfc3339();
let records: Vec<playbook_memory::FailureRecord> = req.failed_names.iter()
.map(|n| playbook_memory::FailureRecord {
city: city.clone(), state: state_code.clone(), name: n.clone(),
reason: req.reason.clone(), timestamp: ts.clone(),
})
.collect();
match state.playbook_memory.mark_failures(records).await {
Ok(added) => Ok(Json(serde_json::json!({ "added": added, "city": city, "state": state_code }))),
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
}
}
async fn playbook_memory_stats( async fn playbook_memory_stats(
State(state): State<VectorState>, State(state): State<VectorState>,
) -> impl IntoResponse { ) -> impl IntoResponse {
@@ -2260,6 +2455,221 @@ async fn playbook_memory_stats(
})) }))
} }
#[derive(Deserialize)]
struct RetirePlaybookRequest {
/// Retire by playbook_id — exact match, single entry. Used for
/// manual operator retirement via the UI.
#[serde(default)]
playbook_id: Option<String>,
/// Retire by scope — city + state required, with a fingerprint
/// that entries must match to survive. Fingerprint mismatch → retire.
/// Use when a schema migration produces a new fingerprint and
/// historical playbooks need to be auto-retired.
#[serde(default)]
city: Option<String>,
#[serde(default)]
state: Option<String>,
#[serde(default)]
current_schema_fingerprint: Option<String>,
/// Human-readable reason stored on the retired entry.
reason: String,
}
/// Phase 25 retirement endpoint. Two modes:
/// {playbook_id, reason} → retire one
/// {city, state, current_schema_fingerprint, reason} → retire all
/// entries in scope whose
/// fingerprint differs
async fn retire_playbook_memory(
State(state): State<VectorState>,
Json(req): Json<RetirePlaybookRequest>,
) -> impl IntoResponse {
if let Some(id) = &req.playbook_id {
return match state.playbook_memory.retire_one(id, &req.reason).await {
Ok(found) => Ok(Json(serde_json::json!({ "mode": "by_id", "retired": if found { 1 } else { 0 } }))),
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
};
}
if let (Some(city), Some(state_code), Some(fp)) = (&req.city, &req.state, &req.current_schema_fingerprint) {
return match state.playbook_memory.retire_on_schema_drift(city, state_code, fp, &req.reason).await {
Ok(n) => Ok(Json(serde_json::json!({ "mode": "schema_drift", "retired": n, "city": city, "state": state_code }))),
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
};
}
Err((StatusCode::BAD_REQUEST,
"supply either {playbook_id, reason} or {city, state, current_schema_fingerprint, reason}".into()))
}
/// Phase 27 — request body for `POST /playbook_memory/revise`. Same
/// shape as a seed request minus `append` (revise is always
/// append-semantics for a specific parent) plus `parent_id`. The new
/// version's `playbook_id` is derived deterministically so callers get
/// the same id back from repeated revises with identical content —
/// useful for idempotent retry paths.
#[derive(Deserialize)]
struct RevisePlaybookRequest {
parent_id: String,
operation: String,
approach: String,
context: String,
endorsed_names: Vec<String>,
#[serde(default)]
schema_fingerprint: Option<String>,
#[serde(default)]
valid_until: Option<String>,
/// Phase 45 — updated doc references. Typically a revise happens
/// BECAUSE docs drifted; pass the new versions seen so the revised
/// entry starts with fresh drift signal.
#[serde(default)]
doc_refs: Option<Vec<playbook_memory::DocRef>>,
}
/// Phase 27 — create a new version of an existing playbook. The parent
/// is marked superseded; the new entry inherits the chain via
/// `parent_id` and carries `version = parent.version + 1`. Errors with
/// 400 on a retired or already-superseded parent (must revise the tip
/// of the chain). Embeds the new text through the same shape as
/// `/seed` so cosine similarity stays comparable across rebuild + seed
/// + revise entries.
async fn revise_playbook_memory(
State(state): State<VectorState>,
Json(req): Json<RevisePlaybookRequest>,
) -> Result<Json<serde_json::Value>, (StatusCode, String)> {
let text = format!(
"{} | {} | {} | fills: {}",
req.operation, req.approach, req.context,
req.endorsed_names.join(", "),
);
let resp = state.ai_client.embed(EmbedRequest { texts: vec![text], model: None })
.await
.map_err(|e| (StatusCode::BAD_GATEWAY, format!("embed revise: {e}")))?;
if resp.embeddings.is_empty() {
return Err((StatusCode::BAD_GATEWAY, "embed returned nothing".into()));
}
let emb: Vec<f32> = resp.embeddings[0].iter().map(|&x| x as f32).collect();
let (city, state_) = {
let after_in = req.operation.split(" in ").nth(1).unwrap_or("");
let mut parts = after_in.splitn(2, ',');
let city = parts.next().map(|s| s.trim().to_string()).filter(|s| !s.is_empty());
let state = parts.next()
.map(|s| s.trim().chars().take_while(|c| c.is_ascii_alphabetic()).collect::<String>())
.filter(|s| !s.is_empty());
(city, state)
};
if city.is_none() || state_.is_none() {
return Err((StatusCode::BAD_REQUEST,
"operation must match 'fill: Role xN in City, ST' shape".into()));
}
// Phase 27 — deterministic pid derived ONLY from content-shaped
// inputs (parent_id + operation + approach + context + sorted
// endorsed_names). Excluding wall-clock ts means two revise calls
// with identical payloads produce the same pid, which is the
// contract the docstring promises. On retry the caller gets back
// the same id and the pre-flight idempotency check below short-
// circuits to the existing entry instead of re-appending.
let mut names_sorted = req.endorsed_names.clone();
names_sorted.sort();
use sha2::{Digest, Sha256};
let mut h = Sha256::new();
h.update(req.parent_id.as_bytes()); h.update(b"|");
h.update(req.operation.as_bytes()); h.update(b"|");
h.update(req.approach.as_bytes()); h.update(b"|");
h.update(req.context.as_bytes()); h.update(b"|");
h.update(names_sorted.join(",").as_bytes());
let bytes = h.finalize();
let pid = format!("pb-rev-{}", bytes.iter().take(8).map(|b| format!("{b:02x}")).collect::<String>());
// Idempotency short-circuit — if this exact pid already exists in
// memory (from a prior successful revise with the same content),
// return it directly rather than re-appending or 400ing on the
// superseded-parent rejection. Walks the parent's chain via
// history() because the parent may itself have been superseded by
// our prior successful call.
let chain = state.playbook_memory.history(&req.parent_id).await;
if let Some(existing) = chain.iter().find(|e| e.playbook_id == pid) {
return Ok(Json(serde_json::json!({
"outcome": {
"parent_id": req.parent_id,
"parent_version": existing.version.saturating_sub(1),
"new_playbook_id": existing.playbook_id,
"new_version": existing.version,
"superseded_at": existing.superseded_at.clone().unwrap_or_default(),
"idempotent_return": true,
},
"entries_after": state.playbook_memory.entry_count().await,
})));
}
let ts = chrono::Utc::now().to_rfc3339();
let new_entry = playbook_memory::PlaybookEntry {
playbook_id: pid.clone(),
operation: req.operation,
approach: req.approach,
context: req.context,
timestamp: ts,
endorsed_names: req.endorsed_names,
city, state: state_,
embedding: Some(emb),
schema_fingerprint: req.schema_fingerprint,
valid_until: req.valid_until,
// Phase 45 — doc_refs may be provided on revise too.
doc_refs: req.doc_refs.clone().unwrap_or_default(),
// revise_entry overwrites version / parent_id / supersession
// from the parent; other fields keep defaults.
..Default::default()
};
let outcome = state.playbook_memory.revise_entry(&req.parent_id, new_entry)
.await
.map_err(|e| (StatusCode::BAD_REQUEST, e))?;
Ok(Json(serde_json::json!({
"outcome": outcome,
"entries_after": state.playbook_memory.entry_count().await,
})))
}
/// Phase 27 — return the full version chain containing `playbook_id`,
/// ordered root → tip. 404 if the id isn't present. The walker is
/// cycle-safe by construction (visited set per direction).
async fn playbook_memory_history(
State(state): State<VectorState>,
Path(playbook_id): Path<String>,
) -> Result<Json<serde_json::Value>, (StatusCode, String)> {
let chain = state.playbook_memory.history(&playbook_id).await;
if chain.is_empty() {
return Err((StatusCode::NOT_FOUND, format!("no playbook with id '{playbook_id}'")));
}
Ok(Json(serde_json::json!({
"playbook_id": playbook_id,
"versions": chain.len(),
"chain": chain,
})))
}
/// Phase 25 status endpoint — reports retirement counts so dashboards
/// can show "N playbooks retired (12 from 2026-05 schema migration)".
/// Phase 27 added `superseded` as a distinct counter.
async fn playbook_memory_status(
State(state): State<VectorState>,
) -> impl IntoResponse {
let (total, retired, superseded, failures) = state.playbook_memory.status_counts().await;
// `active` = entries eligible for boost. Retired and superseded are
// distinct exclusion reasons; subtract both. An entry can in principle
// be both retired AND superseded (e.g. revised then retired) so
// saturating_sub guards against underflow if that pathological case
// ever lands.
let inactive = retired + superseded;
Json(serde_json::json!({
"total": total,
"retired": retired,
"superseded": superseded,
"active": total.saturating_sub(inactive),
"failures": failures,
}))
}
async fn lance_recall_harness( async fn lance_recall_harness(
State(state): State<VectorState>, State(state): State<VectorState>,
Path(index_name): Path<String>, Path(index_name): Path<String>,
@@ -2351,3 +2761,46 @@ async fn lance_build_scalar_index(
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)), Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
} }
} }
#[cfg(test)]
mod extractor_tests {
use super::*;
#[test]
fn extract_target_geo_basic() {
let f = "role = 'Welder' AND city = 'Toledo' AND state = 'OH' AND CAST(availability AS DOUBLE) > 0.5";
assert_eq!(extract_target_geo(f), Some(("Toledo".into(), "OH".into())));
}
#[test]
fn extract_target_geo_missing_state_returns_none() {
let f = "role = 'Welder' AND city = 'Toledo'";
assert_eq!(extract_target_geo(f), None);
}
#[test]
fn extract_target_geo_word_boundary() {
// "civilian" contains "city" as a substring — must not match.
let f = "civilian_rank = 1 AND city = 'Toledo' AND state = 'OH'";
assert_eq!(extract_target_geo(f), Some(("Toledo".into(), "OH".into())));
}
#[test]
fn extract_target_role_basic() {
let f = "role = 'Welder' AND city = 'Toledo'";
assert_eq!(extract_target_role(f), Some("Welder".into()));
}
#[test]
fn extract_target_role_none_when_absent() {
let f = "city = 'Toledo' AND state = 'OH'";
assert_eq!(extract_target_role(f), None);
}
#[test]
fn extract_target_role_multi_word() {
let f = "role = 'Warehouse Associate' AND city = 'Chicago'";
assert_eq!(extract_target_role(f), Some("Warehouse Associate".into()));
}
}
data/_catalog/manifests/07c119c3-2e59-463c-89a8-7980a849b6d6.json
+159
View File
@@ -0,0 +1,159 @@
{
"id": "07c119c3-2e59-463c-89a8-7980a849b6d6",
"name": "workers_500k",
"schema_fingerprint": "71686e497f5ee2fc77d50b13c39a46d89ca3d7673e9a0879e67c2db476764b25",
"objects": [
{
"bucket": "primary",
"key": "datasets/workers_500k.parquet",
"size_bytes": 323794818,
"created_at": "2026-04-17T05:57:41.972944716Z"
}
],
"created_at": "2026-04-17T05:57:41.972945960Z",
"updated_at": "2026-04-17T05:57:41.973093967Z",
"description": "",
"owner": "",
"sensitivity": "pii",
"columns": [
{
"name": "worker_id",
"data_type": "Int64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "name",
"data_type": "Utf8",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "role",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "email",
"data_type": "Utf8",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "phone",
"data_type": "Int64",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "city",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "state",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "zip",
"data_type": "Int64",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "skills",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "certifications",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "archetype",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "reliability",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "responsiveness",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "engagement",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "compliance",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "availability",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "communications",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "resume_text",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
}
],
"lineage": {
"source_system": "csv",
"source_file": "workers_500k.csv",
"ingest_job": "ingest-1776405461972",
"ingest_timestamp": "2026-04-17T05:57:41.972944716Z",
"parent_datasets": []
},
"freshness": null,
"tags": [],
"row_count": 500000,
"last_embedded_at": null,
"embedding_stale_since": null,
"embedding_refresh_policy": null
}
data/_catalog/manifests/443d63f1-b0ed-4d4b-8e5a-ce59c097b97b.json → data/_catalog/manifests/2b8e8b01-a44e-49c5-a103-a1b55898bee1.json
+6 -6
View File
@@ -1,5 +1,5 @@
{ {
"id": "443d63f1-b0ed-4d4b-8e5a-ce59c097b97b", "id": "2b8e8b01-a44e-49c5-a103-a1b55898bee1",
"name": "kb_memory_entries", "name": "kb_memory_entries",
"schema_fingerprint": "15dbebd0abb906577e11cbb73083abeb8961b8c603a2472cafa8c46100c3fb1c", "schema_fingerprint": "15dbebd0abb906577e11cbb73083abeb8961b8c603a2472cafa8c46100c3fb1c",
"objects": [ "objects": [
@@ -7,11 +7,11 @@
"bucket": "primary", "bucket": "primary",
"key": "datasets/kb_memory_entries.parquet", "key": "datasets/kb_memory_entries.parquet",
"size_bytes": 8795, "size_bytes": 8795,
"created_at": "2026-04-17T02:54:43.330897823Z" "created_at": "2026-04-17T05:22:06.356227657Z"
} }
], ],
"created_at": "2026-04-17T02:54:43.330898330Z", "created_at": "2026-04-17T05:22:06.356228933Z",
"updated_at": "2026-04-17T02:54:43.330972517Z", "updated_at": "2026-04-17T05:22:06.356776285Z",
"description": "", "description": "",
"owner": "", "owner": "",
"sensitivity": null, "sensitivity": null,
@@ -97,8 +97,8 @@
"lineage": { "lineage": {
"source_system": "postgresql", "source_system": "postgresql",
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base", "source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
"ingest_job": "pg-stream-1776394483330", "ingest_job": "pg-stream-1776403326356",
"ingest_timestamp": "2026-04-17T02:54:43.330897823Z", "ingest_timestamp": "2026-04-17T05:22:06.356227657Z",
"parent_datasets": [] "parent_datasets": []
}, },
"freshness": null, "freshness": null,
data/_catalog/manifests/2db2d92f-1171-4f2f-a707-ad387b6cd464.json
+117
View File
@@ -0,0 +1,117 @@
{
"id": "2db2d92f-1171-4f2f-a707-ad387b6cd464",
"name": "onboard_live_1776729919",
"schema_fingerprint": "9cdf2fc924bad875e63bd12a82dfb1cd080c5cbba71f671b4a2d41e746df187d",
"objects": [
{
"bucket": "primary",
"key": "datasets/onboard_live_1776729919.parquet",
"size_bytes": 31444,
"created_at": "2026-04-21T00:05:19.717239967Z"
}
],
"created_at": "2026-04-21T00:05:19.717241621Z",
"updated_at": "2026-04-21T00:05:19.717370050Z",
"description": "",
"owner": "",
"sensitivity": "pii",
"columns": [
{
"name": "worker_id",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "name",
"data_type": "Utf8",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "role",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "city",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "state",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "email",
"data_type": "Utf8",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "phone",
"data_type": "Utf8",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "skills",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "certifications",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "availability",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "reliability",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "archetype",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
}
],
"lineage": {
"source_system": "csv",
"source_file": "sample_1776729919.csv",
"ingest_job": "ingest-1776729919717",
"ingest_timestamp": "2026-04-21T00:05:19.717239967Z",
"parent_datasets": []
},
"freshness": null,
"tags": [],
"row_count": 173,
"last_embedded_at": null,
"embedding_stale_since": null,
"embedding_refresh_policy": null
}
data/_catalog/manifests/ab2b610a-cee7-40e1-9dab-c709e2292709.json → data/_catalog/manifests/3178f0da-5985-4731-b343-162dd415f3be.json
+6 -6
View File
@@ -1,5 +1,5 @@
{ {
"id": "ab2b610a-cee7-40e1-9dab-c709e2292709", "id": "3178f0da-5985-4731-b343-162dd415f3be",
"name": "kb_self_reports", "name": "kb_self_reports",
"schema_fingerprint": "76382b028b92bb0b361306dced3d773af0ee8de6aa160cedfc4d649f2860167a", "schema_fingerprint": "76382b028b92bb0b361306dced3d773af0ee8de6aa160cedfc4d649f2860167a",
"objects": [ "objects": [
@@ -7,11 +7,11 @@
"bucket": "primary", "bucket": "primary",
"key": "datasets/kb_self_reports.parquet", "key": "datasets/kb_self_reports.parquet",
"size_bytes": 51134, "size_bytes": 51134,
"created_at": "2026-04-17T02:54:43.335457986Z" "created_at": "2026-04-17T04:14:34.355419647Z"
} }
], ],
"created_at": "2026-04-17T02:54:43.335458749Z", "created_at": "2026-04-17T04:14:34.355420701Z",
"updated_at": "2026-04-17T02:54:43.335536869Z", "updated_at": "2026-04-17T06:06:21.358604060Z",
"description": "", "description": "",
"owner": "", "owner": "",
"sensitivity": null, "sensitivity": null,
@@ -62,8 +62,8 @@
"lineage": { "lineage": {
"source_system": "postgresql", "source_system": "postgresql",
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base", "source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
"ingest_job": "pg-stream-1776394483335", "ingest_job": "pg-stream-1776399274355",
"ingest_timestamp": "2026-04-17T02:54:43.335457986Z", "ingest_timestamp": "2026-04-17T04:14:34.355419647Z",
"parent_datasets": [] "parent_datasets": []
}, },
"freshness": null, "freshness": null,
data/_catalog/manifests/32ee74a0-59b4-4e5b-8edb-70c9347a4bf3.json
+26
View File
@@ -0,0 +1,26 @@
{
"id": "32ee74a0-59b4-4e5b-8edb-70c9347a4bf3",
"name": "successful_playbooks_live",
"schema_fingerprint": "4e28e94650d9f5689e952674aebfd09db70edfcba617713081c0a83bf59121ce",
"objects": [
{
"bucket": "primary",
"key": "datasets/successful_playbooks_live.parquet",
"size_bytes": 284341,
"created_at": "2026-04-22T03:28:28.343840136Z"
}
],
"created_at": "2026-04-20T11:07:57.308050648Z",
"updated_at": "2026-04-22T03:28:28.343843823Z",
"description": "",
"owner": "",
"sensitivity": null,
"columns": [],
"lineage": null,
"freshness": null,
"tags": [],
"row_count": null,
"last_embedded_at": null,
"embedding_stale_since": null,
"embedding_refresh_policy": null
}
data/_catalog/manifests/33544879-fcfa-4458-b782-a4ca8dafed08.json
+159
View File
@@ -0,0 +1,159 @@
{
"id": "33544879-fcfa-4458-b782-a4ca8dafed08",
"name": "workers_100k",
"schema_fingerprint": "1cc2a0caf174aa9bf67b1e2cb7858a312a0c94d604d906f807eb5610864ff70e",
"objects": [
{
"bucket": "primary",
"key": "datasets/workers_100k.parquet",
"size_bytes": 65225188,
"created_at": "2026-04-17T04:54:03.306737596Z"
}
],
"created_at": "2026-04-17T04:54:03.306740279Z",
"updated_at": "2026-04-17T04:54:03.306927852Z",
"description": "",
"owner": "",
"sensitivity": "pii",
"columns": [
{
"name": "worker_id",
"data_type": "Int64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "name",
"data_type": "Utf8",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "role",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "email",
"data_type": "Utf8",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "phone",
"data_type": "Int64",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "city",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "state",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "zip",
"data_type": "Int64",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "skills",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "certifications",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "archetype",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "reliability",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "responsiveness",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "engagement",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "compliance",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "availability",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "communications",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "resume_text",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
}
],
"lineage": {
"source_system": "csv",
"source_file": "workers_100k.csv",
"ingest_job": "ingest-1776401643306",
"ingest_timestamp": "2026-04-17T04:54:03.306737596Z",
"parent_datasets": []
},
"freshness": null,
"tags": [],
"row_count": 100000,
"last_embedded_at": null,
"embedding_stale_since": null,
"embedding_refresh_policy": null
}
data/_catalog/manifests/ce3fe55b-61cd-4ac4-95de-b91f9186d6e3.json → data/_catalog/manifests/35f36c38-5cd3-438e-b93d-7ff7720f0346.json
+5 -11
View File
@@ -1,5 +1,5 @@
{ {
"id": "ce3fe55b-61cd-4ac4-95de-b91f9186d6e3", "id": "35f36c38-5cd3-438e-b93d-7ff7720f0346",
"name": "kb_meta_runs", "name": "kb_meta_runs",
"schema_fingerprint": "68f2c0d7a3ceb0aaa3c17c64900704519c72d213161bc9e5179c42ee53f6d0df", "schema_fingerprint": "68f2c0d7a3ceb0aaa3c17c64900704519c72d213161bc9e5179c42ee53f6d0df",
"objects": [ "objects": [
@@ -7,11 +7,11 @@
"bucket": "primary", "bucket": "primary",
"key": "datasets/kb_meta_runs.parquet", "key": "datasets/kb_meta_runs.parquet",
"size_bytes": 886387, "size_bytes": 886387,
"created_at": "2026-04-17T02:54:43.299467359Z" "created_at": "2026-04-17T05:22:06.314413387Z"
} }
], ],
"created_at": "2026-04-17T02:54:43.299468094Z", "created_at": "2026-04-17T05:22:06.314414515Z",
"updated_at": "2026-04-17T04:23:16.134134852Z", "updated_at": "2026-04-17T06:06:21.300610797Z",
"description": "", "description": "",
"owner": "", "owner": "",
"sensitivity": null, "sensitivity": null,
@@ -80,13 +80,7 @@
"is_pii": false "is_pii": false
} }
], ],
"lineage": { "lineage": null,
"source_system": "postgresql",
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
"ingest_job": "pg-stream-1776399796133",
"ingest_timestamp": "2026-04-17T04:23:16.133868872Z",
"parent_datasets": []
},
"freshness": null, "freshness": null,
"tags": [], "tags": [],
"row_count": 48, "row_count": 48,
data/_catalog/manifests/36b0f141-93a1-4776-beef-b0b8d17a12a9.json
-96
View File
@@ -1,96 +0,0 @@
{
"id": "36b0f141-93a1-4776-beef-b0b8d17a12a9",
"name": "demo_customers",
"schema_fingerprint": "7af9708f2366f790e0ae03db6f681f5a4d4b8af4d1527b091bedfee1be81d304",
"objects": [
{
"bucket": "primary",
"key": "datasets/demo_customers.parquet",
"size_bytes": 4583,
"created_at": "2026-04-17T01:35:01.636480725Z"
}
],
"created_at": "2026-04-17T01:35:01.636481698Z",
"updated_at": "2026-04-17T01:35:01.636776595Z",
"description": "",
"owner": "",
"sensitivity": "pii",
"columns": [
{
"name": "id",
"data_type": "Int32",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "name",
"data_type": "Utf8",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "email",
"data_type": "Utf8",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "city",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "tier",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "lifetime_spend",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "is_active",
"data_type": "Int32",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "signed_up_at",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "notes",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
}
],
"lineage": {
"source_system": "mysql",
"source_file": "dsn: mysql://lh:***@127.0.0.1:3306/lh_demo",
"ingest_job": "scheduled-mysql-1776389701636",
"ingest_timestamp": "2026-04-17T01:35:01.636480725Z",
"parent_datasets": []
},
"freshness": null,
"tags": [],
"row_count": 11,
"last_embedded_at": null,
"embedding_stale_since": null,
"embedding_refresh_policy": null
}
data/_catalog/manifests/564b00ae-cbf3-4efd-aa55-84cdb6d2b0b7.json
+117
View File
@@ -0,0 +1,117 @@
{
"id": "564b00ae-cbf3-4efd-aa55-84cdb6d2b0b7",
"name": "client_workerskjkk",
"schema_fingerprint": "cdfe85348885ddf329e5e6e9bf0e2c75c92d1a86fdb0fd3875ed46e3f93c4d82",
"objects": [
{
"bucket": "primary",
"key": "datasets/client_workerskjkk.parquet",
"size_bytes": 32201,
"created_at": "2026-04-21T00:49:04.623625149Z"
}
],
"created_at": "2026-04-21T00:49:04.623626738Z",
"updated_at": "2026-04-21T00:49:04.623901788Z",
"description": "",
"owner": "",
"sensitivity": "pii",
"columns": [
{
"name": "worker_id",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "name",
"data_type": "Utf8",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "role",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "city",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "state",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "email",
"data_type": "Utf8",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "phone",
"data_type": "Utf8",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "skills",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "certifications",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "availability",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "reliability",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "archetype",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
}
],
"lineage": {
"source_system": "csv",
"source_file": "staffing_roster_sample.csv",
"ingest_job": "ingest-1776732544623",
"ingest_timestamp": "2026-04-21T00:49:04.623625149Z",
"parent_datasets": []
},
"freshness": null,
"tags": [],
"row_count": 180,
"last_embedded_at": null,
"embedding_stale_since": null,
"embedding_refresh_policy": null
}
data/_catalog/manifests/5bab2fac-8bdd-41b7-a993-ed48fecc60b3.json
+117
View File
@@ -0,0 +1,117 @@
{
"id": "5bab2fac-8bdd-41b7-a993-ed48fecc60b3",
"name": "onboard_demo2",
"schema_fingerprint": "274575423f7d2648cea49a628a2e827dbd8822cef1f4eebfcf8e0a5f8602fbf7",
"objects": [
{
"bucket": "primary",
"key": "datasets/onboard_demo2.parquet",
"size_bytes": 7833,
"created_at": "2026-04-20T23:13:36.886222056Z"
}
],
"created_at": "2026-04-20T23:13:36.886223472Z",
"updated_at": "2026-04-20T23:13:36.886426834Z",
"description": "",
"owner": "",
"sensitivity": "pii",
"columns": [
{
"name": "worker_id",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "name",
"data_type": "Utf8",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "role",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "city",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "state",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "email",
"data_type": "Utf8",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "phone",
"data_type": "Utf8",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "skills",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "certifications",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "availability",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "reliability",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "archetype",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
}
],
"lineage": {
"source_system": "csv",
"source_file": "staffing_roster_sample.csv",
"ingest_job": "ingest-1776726816886",
"ingest_timestamp": "2026-04-20T23:13:36.886222056Z",
"parent_datasets": []
},
"freshness": null,
"tags": [],
"row_count": 25,
"last_embedded_at": null,
"embedding_stale_since": null,
"embedding_refresh_policy": null
}
data/_catalog/manifests/8192d934-fc90-46dd-b8bd-c443a4743b19.json
-96
View File
@@ -1,96 +0,0 @@
{
"id": "8192d934-fc90-46dd-b8bd-c443a4743b19",
"name": "demo_customers",
"schema_fingerprint": "7af9708f2366f790e0ae03db6f681f5a4d4b8af4d1527b091bedfee1be81d304",
"objects": [
{
"bucket": "primary",
"key": "datasets/demo_customers.parquet",
"size_bytes": 4519,
"created_at": "2026-04-17T01:00:48.361437263Z"
}
],
"created_at": "2026-04-17T01:00:48.361437860Z",
"updated_at": "2026-04-17T01:00:48.361681724Z",
"description": "",
"owner": "",
"sensitivity": "pii",
"columns": [
{
"name": "id",
"data_type": "Int32",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "name",
"data_type": "Utf8",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "email",
"data_type": "Utf8",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "city",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "tier",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "lifetime_spend",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "is_active",
"data_type": "Int32",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "signed_up_at",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "notes",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
}
],
"lineage": {
"source_system": "mysql",
"source_file": "dsn: mysql://lh:***@127.0.0.1:3306/lh_demo",
"ingest_job": "mysql-stream-1776387648361",
"ingest_timestamp": "2026-04-17T01:00:48.361437263Z",
"parent_datasets": []
},
"freshness": null,
"tags": [],
"row_count": 10,
"last_embedded_at": null,
"embedding_stale_since": null,
"embedding_refresh_policy": null
}
data/_catalog/manifests/8621894c-f9c6-4eea-8e71-e80b9cdc9a36.json
-90
View File
@@ -1,90 +0,0 @@
{
"id": "8621894c-f9c6-4eea-8e71-e80b9cdc9a36",
"name": "demo_customers",
"schema_fingerprint": "7af9708f2366f790e0ae03db6f681f5a4d4b8af4d1527b091bedfee1be81d304",
"objects": [
{
"bucket": "primary",
"key": "datasets/demo_customers.parquet",
"size_bytes": 4583,
"created_at": "2026-04-17T01:34:30.158848713Z"
}
],
"created_at": "2026-04-17T01:34:30.158849587Z",
"updated_at": "2026-04-17T05:56:27.642214035Z",
"description": "",
"owner": "",
"sensitivity": null,
"columns": [
{
"name": "id",
"data_type": "Int32",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "name",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "email",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "city",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "tier",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "lifetime_spend",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "is_active",
"data_type": "Int32",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "signed_up_at",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "notes",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
}
],
"lineage": null,
"freshness": null,
"tags": [],
"row_count": 11,
"last_embedded_at": null,
"embedding_stale_since": null,
"embedding_refresh_policy": null
}
data/_catalog/manifests/8f37454b-27c8-4f61-aeca-8e48070db552.json
-96
View File
@@ -1,96 +0,0 @@
{
"id": "8f37454b-27c8-4f61-aeca-8e48070db552",
"name": "demo_customers",
"schema_fingerprint": "7af9708f2366f790e0ae03db6f681f5a4d4b8af4d1527b091bedfee1be81d304",
"objects": [
{
"bucket": "primary",
"key": "datasets/demo_customers.parquet",
"size_bytes": 4519,
"created_at": "2026-04-17T01:34:00.160615345Z"
}
],
"created_at": "2026-04-17T01:34:00.160616631Z",
"updated_at": "2026-04-17T02:45:58.283076939Z",
"description": "",
"owner": "",
"sensitivity": "pii",
"columns": [
{
"name": "id",
"data_type": "Int32",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "name",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "email",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "city",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "tier",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "lifetime_spend",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "is_active",
"data_type": "Int32",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "signed_up_at",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "notes",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
}
],
"lineage": {
"source_system": "mysql",
"source_file": "dsn: mysql://lh:***@127.0.0.1:3306/lh_demo",
"ingest_job": "scheduled-mysql-1776389700165",
"ingest_timestamp": "2026-04-17T01:35:00.165218923Z",
"parent_datasets": []
},
"freshness": null,
"tags": [],
"row_count": 11,
"last_embedded_at": null,
"embedding_stale_since": null,
"embedding_refresh_policy": null
}
data/_catalog/manifests/14698884-071c-4adb-ae50-cfb8d885656c.json → data/_catalog/manifests/909a1fd1-0924-4a5e-a527-d249a07540a1.json
+7 -13
View File
@@ -1,17 +1,17 @@
{ {
"id": "14698884-071c-4adb-ae50-cfb8d885656c", "id": "909a1fd1-0924-4a5e-a527-d249a07540a1",
"name": "kb_response_cache", "name": "kb_response_cache",
"schema_fingerprint": "c90d7be310b5025d2c4d398cf07692d8d9bb46ed591c0a87b339bafcac9ddeed", "schema_fingerprint": "c90d7be310b5025d2c4d398cf07692d8d9bb46ed591c0a87b339bafcac9ddeed",
"objects": [ "objects": [
{ {
"bucket": "primary", "bucket": "primary",
"key": "datasets/kb_response_cache.parquet", "key": "datasets/kb_response_cache.parquet",
"size_bytes": 8360233, "size_bytes": 8464841,
"created_at": "2026-04-17T02:54:43.145342968Z" "created_at": "2026-04-17T05:22:06.168071783Z"
} }
], ],
"created_at": "2026-04-17T02:54:43.145343876Z", "created_at": "2026-04-17T05:22:06.168072339Z",
"updated_at": "2026-04-17T02:54:43.145629696Z", "updated_at": "2026-04-17T06:06:21.356504054Z",
"description": "", "description": "",
"owner": "", "owner": "",
"sensitivity": null, "sensitivity": null,
@@ -94,16 +94,10 @@
"is_pii": false "is_pii": false
} }
], ],
"lineage": { "lineage": null,
"source_system": "postgresql",
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
"ingest_job": "pg-stream-1776394483145",
"ingest_timestamp": "2026-04-17T02:54:43.145342968Z",
"parent_datasets": []
},
"freshness": null, "freshness": null,
"tags": [], "tags": [],
"row_count": 195, "row_count": 197,
"last_embedded_at": null, "last_embedded_at": null,
"embedding_stale_since": null, "embedding_stale_since": null,
"embedding_refresh_policy": null "embedding_refresh_policy": null
data/_catalog/manifests/9861b817-f571-496b-9d7b-c4d385e51ca5.json
+75
View File
@@ -0,0 +1,75 @@
{
"id": "9861b817-f571-496b-9d7b-c4d385e51ca5",
"name": "sparse_workers",
"schema_fingerprint": "1055d5e17fa3285a1cac7d4c949ae03699cd046af671f22d0d1cf4d0024c66a4",
"objects": [
{
"bucket": "primary",
"key": "datasets/sparse_workers.parquet",
"size_bytes": 7619,
"created_at": "2026-04-17T20:31:24.968688194Z"
}
],
"created_at": "2026-04-17T20:31:24.968689336Z",
"updated_at": "2026-04-17T20:31:24.968969734Z",
"description": "",
"owner": "",
"sensitivity": "pii",
"columns": [
{
"name": "name",
"data_type": "Utf8",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "phone",
"data_type": "Int64",
"sensitivity": "pii",
"description": "",
"is_pii": true
},
{
"name": "role",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "city",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "state",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "notes",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
}
],
"lineage": {
"source_system": "csv",
"source_file": "sparse_workers.csv",
"ingest_job": "ingest-1776457884968",
"ingest_timestamp": "2026-04-17T20:31:24.968688194Z",
"parent_datasets": []
},
"freshness": null,
"tags": [],
"row_count": 200,
"last_embedded_at": null,
"embedding_stale_since": null,
"embedding_refresh_policy": null
}
data/_catalog/manifests/b076cff9-c522-48fc-b892-b7cba7b29c9b.json → data/_catalog/manifests/a2206c79-28d6-40f2-b97b-61a5df5bee06.json
+6 -12
View File
@@ -1,17 +1,17 @@
{ {
"id": "b076cff9-c522-48fc-b892-b7cba7b29c9b", "id": "a2206c79-28d6-40f2-b97b-61a5df5bee06",
"name": "kb_threat_intel", "name": "kb_threat_intel",
"schema_fingerprint": "df1e126046147b3de42086880e10c3501a3a615ecddf336bc24957a24c321241", "schema_fingerprint": "df1e126046147b3de42086880e10c3501a3a615ecddf336bc24957a24c321241",
"objects": [ "objects": [
{ {
"bucket": "primary", "bucket": "primary",
"key": "datasets/kb_threat_intel.parquet", "key": "datasets/kb_threat_intel.parquet",
"size_bytes": 247112, "size_bytes": 247318,
"created_at": "2026-04-17T02:54:43.321496407Z" "created_at": "2026-04-17T04:23:16.149697926Z"
} }
], ],
"created_at": "2026-04-17T02:54:43.321497035Z", "created_at": "2026-04-17T04:23:16.149698245Z",
"updated_at": "2026-04-17T02:54:43.321758360Z", "updated_at": "2026-04-17T05:56:44.994303201Z",
"description": "", "description": "",
"owner": "", "owner": "",
"sensitivity": null, "sensitivity": null,
@@ -227,13 +227,7 @@
"is_pii": false "is_pii": false
} }
], ],
"lineage": { "lineage": null,
"source_system": "postgresql",
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
"ingest_job": "pg-stream-1776394483321",
"ingest_timestamp": "2026-04-17T02:54:43.321496407Z",
"parent_datasets": []
},
"freshness": null, "freshness": null,
"tags": [], "tags": [],
"row_count": 54, "row_count": 54,
data/_catalog/manifests/eb370ff1-c037-476f-8c3f-61b96b3a6046.json → data/_catalog/manifests/a99da7e2-8f42-4ff3-8fa7-679d4e514581.json
+7 -7
View File
@@ -1,17 +1,17 @@
{ {
"id": "eb370ff1-c037-476f-8c3f-61b96b3a6046", "id": "a99da7e2-8f42-4ff3-8fa7-679d4e514581",
"name": "kb_response_cache_history", "name": "kb_response_cache_history",
"schema_fingerprint": "9a05c209a51f9543bd7dc9387695b0c67a6abc135b53eaeec458140712bfcf50", "schema_fingerprint": "9a05c209a51f9543bd7dc9387695b0c67a6abc135b53eaeec458140712bfcf50",
"objects": [ "objects": [
{ {
"bucket": "primary", "bucket": "primary",
"key": "datasets/kb_response_cache_history.parquet", "key": "datasets/kb_response_cache_history.parquet",
"size_bytes": 8292303, "size_bytes": 8396568,
"created_at": "2026-04-17T02:54:43.242879858Z" "created_at": "2026-04-17T04:14:34.260241351Z"
} }
], ],
"created_at": "2026-04-17T02:54:43.242880413Z", "created_at": "2026-04-17T04:14:34.260242474Z",
"updated_at": "2026-04-17T05:56:45.008387335Z", "updated_at": "2026-04-17T06:06:21.334716946Z",
"description": "", "description": "",
"owner": "", "owner": "",
"sensitivity": null, "sensitivity": null,
@@ -69,8 +69,8 @@
"lineage": { "lineage": {
"source_system": "postgresql", "source_system": "postgresql",
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base", "source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
"ingest_job": "pg-stream-1776394483242", "ingest_job": "pg-stream-1776403326260",
"ingest_timestamp": "2026-04-17T02:54:43.242879858Z", "ingest_timestamp": "2026-04-17T05:22:06.260249210Z",
"parent_datasets": [] "parent_datasets": []
}, },
"freshness": null, "freshness": null,
data/_catalog/manifests/a5858f94-267f-4382-ba32-0934e1b984f7.json → data/_catalog/manifests/c2f0acdb-df65-4c21-9633-08af4248c85b.json
+5 -11
View File
@@ -1,5 +1,5 @@
{ {
"id": "a5858f94-267f-4382-ba32-0934e1b984f7", "id": "c2f0acdb-df65-4c21-9633-08af4248c85b",
"name": "kb_lab_trials", "name": "kb_lab_trials",
"schema_fingerprint": "1d5782349402439a7e44efd0ccab9ae64ac3044221adef9e828b60b8bbb44dd5", "schema_fingerprint": "1d5782349402439a7e44efd0ccab9ae64ac3044221adef9e828b60b8bbb44dd5",
"objects": [ "objects": [
@@ -7,11 +7,11 @@
"bucket": "primary", "bucket": "primary",
"key": "datasets/kb_lab_trials.parquet", "key": "datasets/kb_lab_trials.parquet",
"size_bytes": 68221, "size_bytes": 68221,
"created_at": "2026-04-17T02:54:43.423304357Z" "created_at": "2026-04-17T05:22:06.382106104Z"
} }
], ],
"created_at": "2026-04-17T02:54:43.423304931Z", "created_at": "2026-04-17T05:22:06.382107354Z",
"updated_at": "2026-04-17T04:23:16.166692762Z", "updated_at": "2026-04-17T06:06:21.358304760Z",
"description": "", "description": "",
"owner": "", "owner": "",
"sensitivity": null, "sensitivity": null,
@@ -87,13 +87,7 @@
"is_pii": false "is_pii": false
} }
], ],
"lineage": { "lineage": null,
"source_system": "postgresql",
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
"ingest_job": "pg-stream-1776399796166",
"ingest_timestamp": "2026-04-17T04:23:16.166457198Z",
"parent_datasets": []
},
"freshness": null, "freshness": null,
"tags": [], "tags": [],
"row_count": 41, "row_count": 41,
data/_catalog/manifests/03c62234-f9f8-40e9-a27e-d5e09ab2713d.json → data/_catalog/manifests/d668a036-7b50-4845-8194-6a03ef370c8d.json
+5 -11
View File
@@ -1,5 +1,5 @@
{ {
"id": "03c62234-f9f8-40e9-a27e-d5e09ab2713d", "id": "d668a036-7b50-4845-8194-6a03ef370c8d",
"name": "kb_pipeline_runs", "name": "kb_pipeline_runs",
"schema_fingerprint": "c019b81feb58ff2aefe4cbe700056a100e25716f2aa8e8415ba8f20656812f75", "schema_fingerprint": "c019b81feb58ff2aefe4cbe700056a100e25716f2aa8e8415ba8f20656812f75",
"objects": [ "objects": [
@@ -7,11 +7,11 @@
"bucket": "primary", "bucket": "primary",
"key": "datasets/kb_pipeline_runs.parquet", "key": "datasets/kb_pipeline_runs.parquet",
"size_bytes": 1518415, "size_bytes": 1518415,
"created_at": "2026-04-17T02:54:43.281447563Z" "created_at": "2026-04-17T05:22:06.289208160Z"
} }
], ],
"created_at": "2026-04-17T02:54:43.281448143Z", "created_at": "2026-04-17T05:22:06.289209657Z",
"updated_at": "2026-04-17T05:56:44.999492793Z", "updated_at": "2026-04-17T06:06:21.311384739Z",
"description": "", "description": "",
"owner": "", "owner": "",
"sensitivity": null, "sensitivity": null,
@@ -87,13 +87,7 @@
"is_pii": false "is_pii": false
} }
], ],
"lineage": { "lineage": null,
"source_system": "postgresql",
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
"ingest_job": "pg-stream-1776403326289",
"ingest_timestamp": "2026-04-17T05:22:06.289208160Z",
"parent_datasets": []
},
"freshness": null, "freshness": null,
"tags": [], "tags": [],
"row_count": 195, "row_count": 195,
data/_catalog/manifests/7ad27f97-622e-49e3-9c38-327cb2334fa1.json → data/_catalog/manifests/d92c5d57-14f9-4ce0-9c4f-19c6db94a170.json
+7 -13
View File
@@ -1,17 +1,17 @@
{ {
"id": "7ad27f97-622e-49e3-9c38-327cb2334fa1", "id": "d92c5d57-14f9-4ce0-9c4f-19c6db94a170",
"name": "kb_team_runs", "name": "kb_team_runs",
"schema_fingerprint": "d704ee2b9b434774aed2258da2fddcdcbab226547a011ba24d4281253657bdd3", "schema_fingerprint": "d704ee2b9b434774aed2258da2fddcdcbab226547a011ba24d4281253657bdd3",
"objects": [ "objects": [
{ {
"bucket": "primary", "bucket": "primary",
"key": "datasets/kb_team_runs.parquet", "key": "datasets/kb_team_runs.parquet",
"size_bytes": 19185025, "size_bytes": 19293718,
"created_at": "2026-04-17T02:54:43.043742588Z" "created_at": "2026-04-17T05:22:06.071898629Z"
} }
], ],
"created_at": "2026-04-17T02:54:43.043743165Z", "created_at": "2026-04-17T05:22:06.071899235Z",
"updated_at": "2026-04-17T02:54:43.043968762Z", "updated_at": "2026-04-17T05:56:44.992569105Z",
"description": "", "description": "",
"owner": "", "owner": "",
"sensitivity": null, "sensitivity": null,
@@ -108,16 +108,10 @@
"is_pii": false "is_pii": false
} }
], ],
"lineage": { "lineage": null,
"source_system": "postgresql",
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
"ingest_job": "pg-stream-1776394483043",
"ingest_timestamp": "2026-04-17T02:54:43.043742588Z",
"parent_datasets": []
},
"freshness": null, "freshness": null,
"tags": [], "tags": [],
"row_count": 588, "row_count": 590,
"last_embedded_at": null, "last_embedded_at": null,
"embedding_stale_since": null, "embedding_stale_since": null,
"embedding_refresh_policy": null "embedding_refresh_policy": null
data/_catalog/manifests/dab56959-d79b-4c68-a6bd-e5fc487621ff.json
+68
View File
@@ -0,0 +1,68 @@
{
"id": "dab56959-d79b-4c68-a6bd-e5fc487621ff",
"name": "successful_playbooks",
"schema_fingerprint": "30b52caca7b6303852cd648882eae9ed87205b80e5e570f6fda08dc8735d9b06",
"objects": [
{
"bucket": "primary",
"key": "datasets/successful_playbooks.parquet",
"size_bytes": 2540,
"created_at": "2026-04-17T22:25:34.467634304Z"
}
],
"created_at": "2026-04-17T22:25:34.467635066Z",
"updated_at": "2026-04-19T08:44:08.533058251Z",
"description": "",
"owner": "",
"sensitivity": null,
"columns": [
{
"name": "timestamp",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "operation",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "approach",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "result",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "context",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
}
],
"lineage": {
"source_system": "csv",
"source_file": "playbook.csv",
"ingest_job": "ingest-1776588248532",
"ingest_timestamp": "2026-04-19T08:44:08.532686879Z",
"parent_datasets": []
},
"freshness": null,
"tags": [],
"row_count": 29,
"last_embedded_at": null,
"embedding_stale_since": null,
"embedding_refresh_policy": null
}
data/_catalog/manifests/e7304f05-5278-4e17-961a-51f2588fd2aa.json
-243
View File
@@ -1,243 +0,0 @@
{
"id": "e7304f05-5278-4e17-961a-51f2588fd2aa",
"name": "threat_intel",
"schema_fingerprint": "df1e126046147b3de42086880e10c3501a3a615ecddf336bc24957a24c321241",
"objects": [
{
"bucket": "primary",
"key": "datasets/threat_intel.parquet",
"size_bytes": 111130,
"created_at": "2026-03-28T01:14:03.054140697Z"
}
],
"created_at": "2026-03-28T01:14:03.054141294Z",
"updated_at": "2026-04-17T05:56:45.009810081Z",
"description": "",
"owner": "",
"sensitivity": null,
"columns": [
{
"name": "id",
"data_type": "Int32",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "ip",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "threat_level",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "classification",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "confidence",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "summary",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "indicators",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "recommendation",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "pattern",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "attack_type",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "likely_automated",
"data_type": "Boolean",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "country",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "country_code",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "city",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "isp",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "org",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "asn",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "is_proxy",
"data_type": "Boolean",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "is_hosting",
"data_type": "Boolean",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "open_ports",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "blocklist_count",
"data_type": "Int32",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "blocklist_total",
"data_type": "Int32",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "blocklists_blocked",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "reverse_dns",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "traceroute",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "log_count",
"data_type": "Int32",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "banned",
"data_type": "Boolean",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "enriched_at",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "updated_at",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "raw_data",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
}
],
"lineage": {
"source_system": "postgresql",
"source_file": "dsn: postgresql://postgres@127.0.0.1:5432/knowledge_base",
"ingest_job": "pg-stream-1776326353882",
"ingest_timestamp": "2026-04-16T07:59:13.882669337Z",
"parent_datasets": []
},
"freshness": null,
"tags": [],
"row_count": 20,
"last_embedded_at": "2026-04-16T15:08:32.348412159Z",
"embedding_stale_since": null,
"embedding_refresh_policy": null
}
data/_catalog/manifests/765985e6-ea16-46b5-a6c0-ddaff346827f.json → data/_catalog/manifests/eeb0d288-e227-478d-9407-78a43b623c5c.json
+8 -14
View File
@@ -1,5 +1,5 @@
{ {
"id": "765985e6-ea16-46b5-a6c0-ddaff346827f", "id": "eeb0d288-e227-478d-9407-78a43b623c5c",
"name": "kb_meta_pipelines", "name": "kb_meta_pipelines",
"schema_fingerprint": "cabe1d7fc442e1dfcfaabf663509d590c6edc84b445a91acaf0ae68b94aff518", "schema_fingerprint": "cabe1d7fc442e1dfcfaabf663509d590c6edc84b445a91acaf0ae68b94aff518",
"objects": [ "objects": [
@@ -7,14 +7,14 @@
"bucket": "primary", "bucket": "primary",
"key": "datasets/kb_meta_pipelines.parquet", "key": "datasets/kb_meta_pipelines.parquet",
"size_bytes": 32085, "size_bytes": 32085,
"created_at": "2026-04-17T02:54:43.307966396Z" "created_at": "2026-04-17T04:14:34.315211960Z"
} }
], ],
"created_at": "2026-04-17T02:54:43.307966799Z", "created_at": "2026-04-17T04:14:34.315212764Z",
"updated_at": "2026-04-17T04:23:16.140272666Z", "updated_at": "2026-04-17T06:06:21.357779083Z",
"description": "", "description": "",
"owner": "", "owner": "",
"sensitivity": "pii", "sensitivity": null,
"columns": [ "columns": [
{ {
"name": "id", "name": "id",
@@ -26,9 +26,9 @@
{ {
"name": "name", "name": "name",
"data_type": "Utf8", "data_type": "Utf8",
"sensitivity": "pii", "sensitivity": null,
"description": "", "description": "",
"is_pii": true "is_pii": false
}, },
{ {
"name": "data_source", "name": "data_source",
@@ -108,13 +108,7 @@
"is_pii": false "is_pii": false
} }
], ],
"lineage": { "lineage": null,
"source_system": "postgresql",
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
"ingest_job": "pg-stream-1776399796139",
"ingest_timestamp": "2026-04-17T04:23:16.139992290Z",
"parent_datasets": []
},
"freshness": null, "freshness": null,
"tags": [], "tags": [],
"row_count": 6, "row_count": 6,
data/_catalog/manifests/ef6c0acf-de7a-4be0-877e-e0c2889f390f.json
+2 -2
View File
@@ -7,11 +7,11 @@
"bucket": "primary", "bucket": "primary",
"key": "datasets/threat_intel.parquet", "key": "datasets/threat_intel.parquet",
"size_bytes": 247112, "size_bytes": 247112,
"created_at": "2026-04-16T07:59:13.882669337Z" "created_at": "2026-04-19T11:19:40.517865325Z"
} }
], ],
"created_at": "2026-04-16T07:59:13.882669807Z", "created_at": "2026-04-16T07:59:13.882669807Z",
"updated_at": "2026-04-17T06:06:21.359639461Z", "updated_at": "2026-04-19T11:19:40.517873046Z",
"description": "", "description": "",
"owner": "", "owner": "",
"sensitivity": null, "sensitivity": null,
data/_catalog/manifests/f2fda821-2f86-4c81-af58-488d184d8243.json
+47
View File
@@ -0,0 +1,47 @@
{
"id": "f2fda821-2f86-4c81-af58-488d184d8243",
"name": "resumes",
"schema_fingerprint": "a2b2c191412faf317800637da5887d30d749d5ba0222d0a8edd12cde08daeb8a",
"objects": [
{
"bucket": "primary",
"key": "datasets/resumes.parquet",
"size_bytes": 3159,
"created_at": "2026-04-19T11:26:29.607950059Z"
}
],
"created_at": "2026-04-19T11:26:29.607951447Z",
"updated_at": "2026-04-19T11:26:29.608054435Z",
"description": "",
"owner": "",
"sensitivity": null,
"columns": [
{
"name": "doc_id",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "resume_text",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
}
],
"lineage": {
"source_system": "ndjson",
"source_file": "resumes.ndjson",
"ingest_job": "ingest-1776597989607",
"ingest_timestamp": "2026-04-19T11:26:29.607950059Z",
"parent_datasets": []
},
"freshness": null,
"tags": [],
"row_count": 10,
"last_embedded_at": null,
"embedding_stale_since": null,
"embedding_refresh_policy": null
}
data/_catalog/manifests/f429ac10-bc7d-41cf-b30f-a9590760ee32.json
-124
View File
@@ -1,124 +0,0 @@
{
"id": "f429ac10-bc7d-41cf-b30f-a9590760ee32",
"name": "kb_team_runs",
"schema_fingerprint": "d704ee2b9b434774aed2258da2fddcdcbab226547a011ba24d4281253657bdd3",
"objects": [
{
"bucket": "primary",
"key": "datasets/kb_team_runs.parquet",
"size_bytes": 19616539,
"created_at": "2026-04-16T06:07:08.188675427Z"
}
],
"created_at": "2026-04-16T06:07:08.188676317Z",
"updated_at": "2026-04-17T05:56:27.639888971Z",
"description": "",
"owner": "",
"sensitivity": null,
"columns": [
{
"name": "id",
"data_type": "Int32",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "mode",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "prompt",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "config",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "responses",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "models_used",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "created_at",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "archived",
"data_type": "Boolean",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "tags",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "notes",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "quality_score",
"data_type": "Float64",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "score_method",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
},
{
"name": "score_metadata",
"data_type": "Utf8",
"sensitivity": null,
"description": "",
"is_pii": false
}
],
"lineage": {
"source_system": "postgresql",
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
"ingest_job": "pg-stream-1776403326071",
"ingest_timestamp": "2026-04-17T05:22:06.071898629Z",
"parent_datasets": []
},
"freshness": null,
"tags": [],
"row_count": 590,
"last_embedded_at": "2026-04-16T15:08:32.471504656Z",
"embedding_stale_since": "2026-04-17T05:22:06.072418149Z",
"embedding_refresh_policy": null
}
data/datasets/candidates.parquet
BIN
View File
Binary file not shown.
data/datasets/placements.parquet
BIN
View File
Binary file not shown.
docs/CONTROL_PLANE_PRD.md
+457
View File
@@ -0,0 +1,457 @@
# PRD — Universal AI Control Plane
**Status:** Long-horizon architecture target as of 2026-04-22. Lakehouse Phases 0-37 (`docs/PRD.md`) are preserved as the reference implementation and first domain-specific consumer. Phases 38+ (control-plane layers) are sequenced below.
**Current domain: staffing.** The immediate proving ground is the staffing substrate already built — synthetic workers_500k, contracts, emails, SMS drafts, playbook memory. Everything Phase 38-44 ships is validated first against that domain. The DevOps / Terraform / Ansible framing from the original PRD draft stays as a **long-horizon target** — architecture-compatible but not in current scope. See §Long-horizon domains at the bottom.
**Owner:** J
**Cross-read:** `docs/PRD.md` for what's shipped (staffing + AI substrate, 13 crates, ~3M rows). This doc for the layered architecture those pieces now fit into.
---
## Phase Sequencing (Phases 38-44)
Ship each phase before starting the next. Each ends with green tests + docs update.
| Phase | Layer | What ships | Est. LOC | Risk |
|---|---|---|---|---|
| 38 | Layer 1 skeleton | `/v1/chat`, `/v1/usage`, `/v1/sessions` routes forwarding to existing `aibridge` → Ollama. Bot migrates as first consumer. | ~400 | Low — additive, no existing routes touched |
| 39 | Layer 3 adapters | `aibridge::ProviderAdapter` trait; Ollama + one new (OpenRouter). `/v1/chat` routes by config. | ~500 | Low-medium |
| 40 | Layer 2 engine | Rules-based routing (`config/routing.toml`), fallback chains, cost gating. Add Gemini + Claude adapters. | ~600 | Medium |
| 41 | Profile split | Separate Retrieval / Memory / Execution / Observer profiles; Phase 17 backward-compat. Absorbs Phase 37 hot-swap-async. | ~300 | Medium |
| 42 | Truth Layer | New `crates/truth`; Terraform/Ansible schemas; `/v1/context` serves rules to router + observer. | ~700 | Medium |
| 43 | Validation pipeline | Syntax/lint/dry-run/policy gates per output type. Plugs into Layer 5 execution loop. | ~400 | Medium |
| 44 | Caller migration | All internal callers route through `/v1/chat`. Direct sidecar access deprecated. | ~200 | Low |
**Total ≈3100 LOC.** Phase 37 (hot-swap async) folds into Phase 41 — it's an Execution-Profile activation concern.
---
## Phase 38 — Universal API Skeleton
**Goal:** OpenAI-compatible `/v1/*` surface exists and forwards to existing aibridge → Ollama. Nothing about multi-provider yet — just the SHAPE, so every downstream piece (adapters, routing, usage accounting) has a surface to plug into.
**Ships:**
- `crates/gateway/src/v1/mod.rs` — router + `/v1/chat`, `/v1/usage`, `/v1/sessions`
- `crates/gateway/src/v1/ollama.rs` — shape adapter (OpenAI chat ↔ existing aibridge `GenerateRequest`)
- One-line `nest("/v1", ...)` in `crates/gateway/src/main.rs`
- Unit test: `POST /v1/chat` roundtrips through mocked provider
**Gate:**
- `curl -X POST localhost:3100/v1/chat -d '{"model":"qwen3.5:latest","messages":[{"role":"user","content":"hi"}]}'` returns valid OpenAI-shape response.
- `GET localhost:3100/v1/usage` returns `{requests, prompt_tokens, completion_tokens, total_tokens}`.
- `GET localhost:3100/v1/sessions` returns `{data:[]}` (stub; real impl Phase 41).
- `cargo test -p gateway` green.
**Non-goals (explicit):** streaming, tool calls, function calling, session state, multi-provider, fallback, cost gating.
**Risk:** Low — additive, doesn't touch existing routes. Worst case: `/v1/*` returns 502 and we fix the adapter. No existing caller affected.
---
## Phase 39 — Provider Adapter Refactor
**Goal:** `aibridge` grows a `ProviderAdapter` trait. Ollama implementation wraps existing sidecar code. One new provider lands as proof: **OpenRouter** (simplest — it's OpenAI-compatible, so adapter is mostly passthrough).
**Ships:**
- `crates/aibridge/src/provider.rs` — `ProviderAdapter` trait with `chat()` + `embed()` + `unload()` methods
- `crates/aibridge/src/providers/ollama.rs` — existing sidecar code moved behind the trait
- `crates/aibridge/src/providers/openrouter.rs` — new, HTTP client to `openrouter.ai/api/v1/chat/completions`
- `config/providers.toml` — provider registry (name, base_url, auth, default_models)
- `/v1/chat` routes by `model` field: prefix match (e.g. `openrouter/anthropic/claude-3.5-sonnet` → OpenRouter; bare names → Ollama)
**Gate:**
- `/v1/chat` with `model: "qwen3.5:latest"` hits Ollama → green
- `/v1/chat` with `model: "openrouter/openai/gpt-4o-mini"` hits OpenRouter (key from secrets.toml) → green
- Neither call leaks provider-specific fields upward. Response is always the `/v1/chat` shape.
**Non-goals:** Fallback chain (Phase 40), cost gating (Phase 40), Gemini/Claude adapters (Phase 40).
**Risk:** Low-medium. The trait extraction is mostly a rearrange; OpenRouter is thin. Biggest risk is secret-loading conventions — `SecretsProvider` is already in place, so reuse that path.
---
## Phase 40 — Routing & Policy Engine + Observability Recovery
**Goal:** Replace hardcoded T1-T5 routing with a rules engine. Add Gemini + Claude adapters. Cost gating enforced at router level. **Reinstate Langfuse + Gitea MCP** — recovery of the observability + repo-ops stack J built previously (see `project_lost_stack` memory).
**Ships — routing:**
- `crates/aibridge/src/routing.rs` — rules engine (match on: task type, token budget, previous attempt failures, profile ID)
- `config/routing.toml` — rules in TOML (human-editable, hot-reloadable)
- `crates/aibridge/src/providers/gemini.rs` — `generativelanguage.googleapis.com` adapter
- `crates/aibridge/src/providers/claude.rs` — `api.anthropic.com` adapter
- Fallback chain support: if primary returns 5xx or times out, try next in chain
- Cost gate: per-request budget + daily budget per-provider
**Ships — observability (was lost, now restored):**
- **Langfuse** self-hosted via Docker Compose. Single source of truth for every LLM call trace: prompt / response / tokens / cost / latency / provider / fallback chain / profile used. UI at `localhost:3000`. Keys in `/etc/lakehouse/secrets.toml`.
- `crates/aibridge/src/langfuse.rs` — thin fire-and-forget trace emitter. Every `/v1/chat` call spawns a background task that POSTs to `langfuse/api/public/ingestion`. Non-blocking: trace failures never affect response.
- **Langfuse → observer pipe** — `mcp-server/langfuse_bridge.ts` or similar. Polls Langfuse's trace API at interval, forwards completed traces to observer `:3800/event` with `source: "langfuse"`. KB now sees cost/latency deltas per model, not just outcome deltas.
- **Gitea MCP reconnect** — the MCP server binary still installed at `/home/profit/.bun/install/cache/gitea-mcp@0.0.10/` gets wired into `mcp-server/index.ts` tool registry. Agents can open PRs, comment on issues, list commits via named tools. Closes Phase 28's repo-ops gap.
**Gate:**
- Rule like "local models for simple JSON emitters, cloud for reasoning" fires correctly by task type
- Primary fails → fallback provider hits, response still matches `/v1/chat` shape
- Daily budget hit → subsequent requests return 429 with clear retry-at header
- `/v1/usage` reports per-provider breakdown
- **Every `/v1/chat` call appears in Langfuse UI** with correct prompt, response, latency, token count within 2 seconds of the request completing
- **Langfuse → observer pipe** delivers trace deltas to KB: `GET :3800/stats?source=langfuse` shows non-zero count after a few scenarios run
- **Gitea MCP tools callable** — `list_prs`, `open_pr`, `comment_on_issue` exposed in `mcp-server/index.ts`, verifiable via a quick agent scenario
**Non-goals:** Retrieval Profile split (Phase 41), Truth Layer (Phase 42). Langfuse self-hosted UI customization / SSO.
**Risk:** Medium. Multi-provider auth + cost tracking is cross-cutting; Langfuse adds 4-5 Docker containers (PostgreSQL, ClickHouse, Redis, web, worker). Mitigation: every provider call wrapped in a single `dispatch()` function so observability flows through one point; Langfuse Docker Compose is their supported deployment path, well-tested.
---
## Phase 41 — Profile System Expansion (+ Phase 37 hot-swap async folded in)
**Goal:** The existing `ModelProfile` (Phase 17) becomes **ExecutionProfile**. Three new profile types land alongside. Profile activation is async — returns job_id, work runs in background (Phase 37 deliverable).
**Ships:**
- `crates/shared/src/profiles/` — `ExecutionProfile`, `RetrievalProfile`, `MemoryProfile`, `ObserverProfile`
- `crates/catalogd` gains per-profile-type CRUD endpoints (`/catalog/profiles/retrieval`, etc.)
- `crates/vectord/src/activation.rs` — `ActivationTracker` with background-job pattern (Phase 37 content)
- `POST /vectors/profile/{id}/activate` returns 202 + job_id, polling at `GET /vectors/profile/jobs/{id}`
- Single-flight guard: refuse new activation if one is pending/running
- Backward compat: `ModelProfile` still loads, aliased to ExecutionProfile
**Gate:**
- Activate a profile → returns 202 in <100ms → job completes in background → `/vectors/profile/jobs/{id}` shows progress + final report
- `tests/multi-agent/run_stress.ts` Phase 3 (hot-swap stress) passes (was SKIPPED)
- Retrieval + Memory + Observer profiles can be created independently of Execution profile
**Non-goals:** Truth Layer (Phase 42), validation (Phase 43), caller migration (Phase 44).
**Risk:** Medium. Schema change + async refactor. Mitigation: `#[serde(default)]` on all new fields; existing profiles load unchanged.
---
## Phase 42 — Truth Layer (staffing rules first)
**Goal:** New `crates/truth` crate holds immutable task-class constraints. Served via `/v1/context` to router and observer. No layer can override truth. **Staffing rules ship first**; Terraform/Ansible rule shapes are scaffolded but unpopulated until the long-horizon phase.
**Ships:**
- `crates/truth/src/lib.rs` — `TruthStore` with schema loading (TOML/YAML rules)
- `crates/truth/src/staffing.rs` — staffing rule shapes:
- Worker eligibility (active status, not blacklisted for client, geo match, role match, availability window)
- Contract invariants (deadline present, role/count/city/state populated, budget_per_hour_max ≥ 0)
- PII handling (redaction rules on fields tagged `PII` before any cloud call — covers existing Phase 10 sensitivity tags)
- Client blacklist enforcement (auto-applied before any fill proposal)
- Fill requirements (endorsed_names count matches target_count, no duplicate worker_ids within a single fill)
- `crates/truth/src/devops.rs` — **scaffold only**: empty rule struct for Terraform/Ansible, populated in the long-horizon phase. Keeps the dispatcher signature stable so no refactor needed later.
- `truth/` dir at repo root — rule files, versioned in git
- `/v1/context` endpoint — returns applicable rules for a task class (`staffing.fill`, `staffing.rescue`, `staffing.sms_draft`, etc.)
- Router consults truth before dispatching: if task violates a rule, hard-fail with structured error + rule citation (matches existing Phase 13 access-control pattern)
**Gate:**
- Submit a fill proposal where a worker is client-blacklisted — router returns 422 + rule citation, no cloud tokens burned
- Submit a fill with `endorsed_names.length != target_count` — 422 before dispatch
- Observer cannot promote a correction that violates truth (rejected at router gate)
- PII redaction verified: SSN / salary fields stripped from prompts before cloud calls
- Truth reload is explicit (no file-watch hot reload in this phase)
**Non-goals:** Validation execution (Phase 43), policy learning / evolution (deferred), actual Terraform/Ansible rules (long-horizon phase).
**Risk:** Medium. Domain-specific rule enumeration takes discovery — start with a minimal rule set (5-10 staffing rules, derived from existing Phase 10-13 work) and grow organically as real fills surface edge cases.
---
## Phase 43 — Validation Pipeline (staffing outputs first)
**Goal:** Staffing outputs run through schema / completeness / consistency / policy gates. Plug into Layer 5 execution loop — failure triggers observer-correction iteration. This is where the **0→85% pattern reproduces on real staffing tasks** — the iteration loop with validation in place is what made small models successful.
**Ships:**
- `crates/validator/src/lib.rs` — `Validator` trait: `validate(artifact) -> Result<Report, ValidationError>` + `Artifact` enum over output types
- `crates/validator/src/staffing/fill.rs` — fill-proposal validator:
- Schema compliance (propose_done shape matches `{fills: [{candidate_id, name}]}`)
- Completeness (endorsed count == target_count)
- Worker existence (every candidate_id present in workers_500k via SQL lookup)
- Status check (every worker has status=active, not_on_client_blacklist)
- Geo/role match (worker city/state/role matches contract)
- `crates/validator/src/staffing/email.rs` — generated email/SMS drafts:
- Schema (TO/BODY fields present)
- Length (SMS ≤ 160 chars; email subject ≤ 78 chars)
- PII absence (no SSN / salary leaked into outgoing text)
- Worker-name consistency (name in message matches worker record)
- `crates/validator/src/staffing/playbook.rs` — sealed playbook:
- Operation format (`fill: Role xN in City, ST`)
- endorsed_names non-empty, ≤ target_count × 2
- fingerprint populated (Phase 25 validity window requirement)
- `crates/validator/src/devops.rs` — **scaffold only**: stubbed Terraform/Ansible validators (`terraform validate`, `ansible-lint`) for the long-horizon phase
- Task execution loop in gateway: generate → validate → if fail, observer correction + retry (bounded by `max_iterations=3`)
- Validation results logged to observer (`data/_observer/ops.jsonl`) + KB (`data/_kb/outcomes.jsonl`)
**Gate:**
- Generate a fill proposal → validator catches a phantom worker_id → observer + cloud rescue propose correction → retry → green. This reproduces the 0→85% pattern on the live staffing pipeline.
- `/v1/usage` shows iteration count per task, provider fallback chain, and tokens-per-iteration. Cost attribution per task class visible.
- Reproduces 14× citation-lift finding from Phase 19 refinement on similar geos after validation gates.
**Non-goals:** Caller migration (Phase 44), Terraform/Ansible wired validation (long-horizon).
**Risk:** Medium. Validation shapes have to match actual executor outputs; mitigation is using real scenario runs as test fixtures (we have ~100 of them in `tests/multi-agent/playbooks/`).
---
## Phase 44 — Caller Migration + Direct-Provider Deprecation
**Goal:** Every internal LLM caller routes through `/v1/chat`. Direct sidecar / direct Ollama / direct OpenAI calls are removed or explicitly deprecated with a warning.
**Ships:**
- `aibridge::AiClient` becomes a thin `/v1/chat` client (was direct-to-sidecar)
- `crates/vectord::agent` (autotune): routes through `/v1`
- `crates/vectord::autotune`: routes through `/v1`
- `tests/multi-agent/agent.ts::generate()`: routes through `/v1`
- `bot/propose.ts`: routes through `/v1` (already proposed as Phase 38's test consumer, formalized here)
- Lint rule / grep pre-commit hook: no `fetch.*:3200/generate` outside the provider adapters
**Gate:**
- `grep -r "localhost:3200/generate\|/api/generate"` returns only adapter files + deprecation shims
- `/v1/usage` accounts for every LLM call in the system within a 1-minute window after hitting a fresh scenario
- Full scenario passes end-to-end without any caller bypassing `/v1/*`
**Non-goals:** New features. This phase is purely mechanical migration.
**Risk:** Low. Mechanical. Tests catch regressions.
---
## Phase 45 — Doc-drift detection + context7 integration
**Goal:** Playbooks know which external docs they were written against. When those docs change (Docker adds a feature, npm lib goes major, Terraform renames a resource), the playbook is automatically flagged. Small models never run confidently-outdated procedures — the drift signal reaches them before the next execution does.
**Why this phase exists at all:** The 0→85% thesis depends on the hyperfocus lane staying valid. External doc drift invalidates the lane silently — popular playbooks can compound the wrong way, accumulating boost while growing more wrong. Phase 25 already retires playbooks on *internal* schema drift; Phase 45 is the same mechanism against *external* doc drift. This is the completion of the learning loop, not an optional add-on.
**Ships:**
- `shared::types::DocRef` — `{ tool: String, version_seen: String, snippet_hash: Option<String>, source_url: Option<String>, seen_at: DateTime<Utc> }`
- `PlaybookEntry.doc_refs: Vec<DocRef>` — `#[serde(default)]` so pre-Phase-45 entries load as empty vec
- `/vectors/playbook_memory/seed` + `/revise` accept `doc_refs` in the request body
- `/vectors/playbook_memory/doc_drift/check/{id}` — manual drift check: looks up each `doc_refs[]` entry via the context7 bridge, returns per-tool `{version_seen, version_current, drifted: bool}` plus overall verdict
- `/vectors/playbook_memory/doc_drift/scan` — batch scan across all active playbooks (scheduled path for Phase 45.2)
- `mcp-server/context7_bridge.ts` — Bun HTTP bridge. Exposes `GET /docs/:tool/version` + `GET /docs/:tool/:version/diff?since=X` against the installed context7 MCP plugin. Gateway calls this over localhost.
- `PlaybookMemory::compute_boost_for_filtered_with_role` — excludes entries where `doc_drift_flagged_at.is_some() && doc_drift_review.is_none()` (same rule as retired + superseded)
- Overview model synthesis writes `data/_kb/doc_drift_corrections.jsonl` per detected drift: `{playbook_id, tool, version_seen, version_current, diff_summary, recommended_action, generated_at}`
- Human-in-the-loop re-seal path: `/vectors/playbook_memory/doc_drift/resolve/{id}` — marks reviewed, optionally triggers `revise_entry` if procedure changed
**Gate:**
- Seal a playbook referencing Docker 24.x → doc_refs captured. Bump Docker version behind the scenes → `/doc_drift/check/{id}` returns `drifted: true, from: 24.0.7, to: 25.0.1, summary: "..."`. The boosted playbook count on next `/vectors/hybrid` query drops by 1 (drift-flagged skipped).
- `doc_drift_corrections.jsonl` contains the overview model's synthesis for the drift with at least: summary of change, recommended action, cost/impact estimate.
- Human calls `/doc_drift/resolve/{id}` after reviewing → playbook returns to active boost pool (or supersedes via Phase 27 if procedure materially changed).
- Unit tests: DocRef serde default (legacy entries load as empty), drift check against mocked context7 bridge, boost exclusion when drifted+unreviewed.
**Non-goals (explicit):**
- Automatic re-seal without human review. Drift-detection → flag, not silent rewrite.
- Cross-playbook propagation of one drift diagnosis. Each playbook reviewed individually (aggregation later if warranted).
- Generating the updated procedure. T3 *suggests*; human or separate bot (see `bot/`) *writes*.
**Risk:** Medium. The context7 bridge is new infrastructure (Bun ↔ context7 MCP plugin ↔ HTTP shape for gateway consumption). Mitigation: context7 plugin is already installed; its MCP tools return structured JSON; the bridge is thin adapter code. Start with single-tool drift check (Docker) before broadening.
---
## Long-horizon domains (not in current phase sequence)
The architecture was drafted with DevOps execution (Terraform, Ansible) as the eventual target. **That remains aspirational, not current scope** — we don't start wiring `terraform validate` / `ansible-lint` until the staffing domain proves the six-layer architecture at scale.
What "proves at scale" means concretely:
- Phases 38-44 all shipped against staffing, green tests
- Live staffing pipeline handles **multiple concurrent contracts** with emails + SMS + indexed playbooks via `/v1/*`
- Observed **iteration success lift** (the 0→85% pattern) reproduced on varied staffing scenarios, not just the original proof-of-concept
- Token + cost accounting stable across provider fallback chains under real load
- Truth Layer rules prevent real fill errors before cloud burn (not just theoretical)
When staffing hits that bar, the DevOps domain lights up by:
- Populating `crates/truth/src/devops.rs` with real Terraform/Ansible rule shapes
- Populating `crates/validator/src/devops.rs` with `terraform validate` / `ansible-lint` shell-out
- Adding DevOps task classes to `/v1/context` rule lookup
- No architectural changes needed — the dispatcher, router, and execution loop stay identical.
Other candidate long-horizon domains (same pattern):
- Code generation tasks (validation via `cargo check` / `bun test`)
- SQL query generation (validation via EXPLAIN + schema compliance)
- Data pipeline definitions (validation via lineage check + schema compliance)
None of these are in the current roadmap. **Staffing first, production-proven, then expand.**
---
## 1. Purpose
Design and implement a universal AI control-plane API that enables:
- **deterministic high-stakes task execution** — the immediate domain is staffing fills (contracts, workers, emails, SMS) at scale; the same architecture extends later to DevOps (Terraform, Ansible) without redesign
- iterative capability amplification via observer loops
- hybrid local + cloud model orchestration
- structured knowledge + memory + playbook reuse
- controlled improvement over time through validated iteration
The system prioritizes **validated pipeline success over raw model intelligence**.
### Current scope — staffing at scale
The architecture must make the already-built staffing substrate reliably answer millions of inputs: pull real data, graph it across contracts, handle multiple concurrent contracts, index emails + SMS + playbooks via the hybrid SQL+vector method, and get **faster and better each iteration** via the feedback loops (Phase 19 playbook boost, Phase 22 KB pathway recommender, Phase 24 observer, Phase 26 Mem0 upsert).
DevOps is an eventual domain — see §Long-horizon domains.
## 2. Core Objectives
### 2.1 Functional Goals
- Provide a single universal API for all AI interactions
- Support multi-provider routing (local, flat-rate, token-based)
- Enable iterative execution loops with observer correction
- Store and reuse successful execution playbooks
- Integrate: S3-based knowledge storage, LanceDB retrieval/indexing, Mem0 memory layer, MCP tool ecosystem
### 2.2 Non-Functional Goals
- Deterministic behavior under constrained execution
- Full observability and cost accounting
- Safe DevOps execution (no uncontrolled mutation)
- Profile-driven routing and execution
- Reproducibility of successful runs
## 3. System Architecture
### 3.1 Layer Overview
**Layer 1 — Universal API**
Single entry point for all applications. Endpoints:
- `/v1/chat`
- `/v1/respond`
- `/v1/tools`
- `/v1/context`
- `/v1/usage`
- `/v1/sessions`
All programs must use this layer. No direct provider calls allowed.
**Layer 2 — Routing & Policy Engine**
Responsibilities: provider selection, fallback logic, cost gating, premium access control, profile enforcement.
Routing based on: task type, constraints, execution profile, system health.
**Layer 3 — Provider Adapter Layer**
Normalizes all providers: Ollama (local), OpenRouter, Gemini (direct), Claude (direct or routed), future providers.
Guarantee: no provider-specific logic leaks upward.
**Layer 4 — Knowledge & Memory Plane**
- Knowledge (S3 + LanceDB): raw documents, processed chunks, embeddings, index profiles
- Memory (Mem0): extracted facts, entity-linked memory, session-aware retrieval
- Playbooks: successful execution traces, reusable patterns, correction strategies
**Layer 5 — Execution Loop**
Each task runs through: Retrieval → Planning → Generation → Validation → Observer feedback → Iteration (if needed).
**Layer 6 — Observability & Accounting**
Every request logs: tokens (input/output), cost, latency, provider, fallback chain, profile used, iteration delta.
## 4. Execution Model
### 4.1 Iterative Loop
Each task follows: **Attempt → Validate → Observe → Adjust → Retry**
Constraints:
- max iterations (default: 3)
- minimum improvement threshold
- cost ceiling per task
### 4.2 Observer Role
Observer can: analyze failure, suggest corrections, recommend profile changes.
Observer cannot: modify truth layer, auto-promote changes, override constraints.
### 4.3 Cloud Escalation
Cloud models (Gemini, Claude) are used for: structural correction, reasoning gaps, complex decomposition.
They are not used for: brute-force retries, bulk execution.
## 5. Profile System
### 5.1 Profile Types
- **Retrieval Profile** — chunking strategy, embedding method, reranking rules
- **Memory Profile** — memory weighting, context injection rules
- **Execution Profile** — allowed providers, tool access, risk level
- **Observer Profile** — mutation aggressiveness, iteration strategy
### 5.2 Profile Constraints
- only one major profile change per iteration
- profiles must produce measurable deltas
- promotion requires repeated success
## 6. Truth Layer (Critical)
Defines non-negotiable constraints:
- Terraform rules
- Ansible structure requirements
- security policies
- organization standards
Rules:
- immutable at runtime
- referenced by all layers
- cannot be overridden by observer
## 7. Playbook System
### 7.1 Playbook Definition
Each successful run produces: task class, context used, steps executed, tools used, output artifacts, validation results, cost/latency, success score.
### 7.2 Playbook Lifecycle
- created on success
- reused for similar tasks
- decayed over time
- pruned if ineffective
## 8. Validation System
All DevOps outputs must pass: syntax validation, linting, dry-run, policy compliance.
Failure → iteration continues or task fails.
## 9. MCP Integration
MCP servers provide: tools, external data, execution capabilities.
All MCP outputs must be: normalized, validated, schema-compliant.
No direct MCP output reaches the model.
## 10. Token Accounting & Budget Control
Each request tracks: input tokens, output tokens, retries, fallback cost.
Policies: premium providers gated, cost ceilings enforced, per-task budget limits.
## 11. Failure Handling
**Recoverable failures:** bad decomposition, missing steps, weak retrieval → observer + iteration.
**Hard failures:** missing truth data, invalid task classification, unsafe execution → termination + error report.
## 12. Success Criteria
A task is successful only if:
- output is valid
- all validators pass
- no policy violations
- result is reproducible
- cost within limits
## 13. Key Risks & Mitigations
- **Observer drift** → bounded authority, confidence tracking
- **Memory poisoning** → validation layer, memory weighting
- **Cost explosion** → token accounting, iteration caps
- **Retrieval errors** → post-retrieval validation, profile tuning
docs/DECISIONS.md
+5
View File
@@ -94,3 +94,8 @@
**Date:** 2026-04-16 **Date:** 2026-04-16
**Decision:** Keep Parquet + binary-blob vectors + in-RAM HNSW as the primary vector backend. Add Lance as a second backend available per-profile for workloads where Lance wins architecturally. Per-profile `vector_backend: Parquet | Lance` field becomes part of Phase 17 model profiles. Implementation kicks off via the standalone `crates/lance-bench` crate and is promoted into `vectord::lance_store` when the API stabilizes. **Decision:** Keep Parquet + binary-blob vectors + in-RAM HNSW as the primary vector backend. Add Lance as a second backend available per-profile for workloads where Lance wins architecturally. Per-profile `vector_backend: Parquet | Lance` field becomes part of Phase 17 model profiles. Implementation kicks off via the standalone `crates/lance-bench` crate and is promoted into `vectord::lance_store` when the API stabilizes.
**Rationale:** Head-to-head benchmark on the 100K × 768d `resumes_100k_v2` index (see `docs/ADR-019-vector-storage.md` for the full scorecard). Parquet+HNSW wins current-scale search latency by 2.55× (873us vs 2229us p50). Lance wins index build time by 14× (16s vs 230s), random row access by 112× (311us vs ~35ms full-file scan), and append speed structurally (0.08s vs full Parquet rewrite). Neither strictly dominates — the dual-use PRD framing (staffing + LLM brain) means both workloads exist in the same system. Keeps ADR-008's "Parquet is the format" principle intact for dataset tables; adds Lance as a purpose-built vector-tier option without discarding the tuned HNSW stack. **Rationale:** Head-to-head benchmark on the 100K × 768d `resumes_100k_v2` index (see `docs/ADR-019-vector-storage.md` for the full scorecard). Parquet+HNSW wins current-scale search latency by 2.55× (873us vs 2229us p50). Lance wins index build time by 14× (16s vs 230s), random row access by 112× (311us vs ~35ms full-file scan), and append speed structurally (0.08s vs full Parquet rewrite). Neither strictly dominates — the dual-use PRD framing (staffing + LLM brain) means both workloads exist in the same system. Keeps ADR-008's "Parquet is the format" principle intact for dataset tables; adds Lance as a purpose-built vector-tier option without discarding the tuned HNSW stack.
## ADR-020: `register()` is idempotent by name with a schema-fingerprint gate
**Date:** 2026-04-19
**Decision:** `catalogd::Registry::register(name, fingerprint, objects)` is idempotent on `name`. If no manifest for `name` exists, create one. If one exists with the same `schema_fingerprint`, reuse its `DatasetId`, replace `objects`, bump `updated_at`, and write through. If one exists with a different `schema_fingerprint`, reject with `409 Conflict` (HTTP) / `FAILED_PRECONDITION` (gRPC). A one-shot operator endpoint `POST /catalog/dedupe` collapses any pre-existing duplicates (preferring the manifest with a non-null `row_count`, then the most recently updated).
**Rationale:** Registry was keyed by surrogate `DatasetId` with no uniqueness constraint on `name`, so every caller that re-registered (re-ingest, external cron, gRPC retry) silently created a parallel manifest pointing at the same parquet — accumulating 308× `successful_playbooks` in live state before detection. The fingerprint gate turns re-ingest into an explicit no-op (matching PRD invariant #5 "ingestd is idempotent — re-ingesting the same file is a no-op") while forcing schema drift to be visible instead of silently clobbering. 409 status separates policy rejections from server errors, which matters for the Phase 12 tool-consumer ecosystem. Concurrency: the write lock is held across the storage write to close the check→insert TOCTOU window; serializing registers is acceptable because registers-per-second is low on the ingest path. Audit: idempotent-register events are visible as bumps to the stored manifest's `updated_at` field and in `catalogd` tracing output (tracing is non-durable, operator view only); `DedupeReport` is the return-value audit for cleanup runs. No event-journal entries are emitted — ADR-012 scopes the journal to row-level mutations, not catalog-manifest operations.
docs/PHASES.md
+89 -6
View File
@@ -188,7 +188,7 @@
- Endpoint: `POST /catalog/datasets/by-name/{name}/tombstone` accepting `{row_key_column, row_key_values[], actor, reason}`; companion `GET` lists active tombstones - Endpoint: `POST /catalog/datasets/by-name/{name}/tombstone` accepting `{row_key_column, row_key_values[], actor, reason}`; companion `GET` lists active tombstones
- `queryd::context::build_context` wraps tombstoned tables: raw goes to `__raw__{name}`, public name becomes a DataFusion view with `WHERE CAST(col AS VARCHAR) NOT IN (...)` filter - `queryd::context::build_context` wraps tombstoned tables: raw goes to `__raw__{name}`, public name becomes a DataFusion view with `WHERE CAST(col AS VARCHAR) NOT IN (...)` filter
- End-to-end on candidates: tombstone 3 IDs, COUNT drops 100,000 → 99,997, specific WHERE returns empty, AiView candidates_safe transitively excludes them too, restart preserves all tombstones - End-to-end on candidates: tombstone 3 IDs, COUNT drops 100,000 → 99,997, specific WHERE returns empty, AiView candidates_safe transitively excludes them too, restart preserves all tombstones
- Limits / not in MVP: physical compaction (Phase 8 doesn't yet read tombstones during merge); journal integration (tombstones don't yet emit Phase 9 mutation events — covered by audit fields on the tombstone itself) - Limits / not in MVP: journal integration (tombstones don't yet emit Phase 9 mutation events — covered by audit fields on the tombstone itself). Physical compaction integration was closed by Phase E.2 below.
- [x] Phase D: AI-safe views — 2026-04-16 - [x] Phase D: AI-safe views — 2026-04-16
- `shared::types::AiView` — name, base_dataset, columns whitelist, optional row_filter, column_redactions - `shared::types::AiView` — name, base_dataset, columns whitelist, optional row_filter, column_redactions
- `shared::types::Redaction` — Null | Hash | Mask { keep_prefix, keep_suffix } - `shared::types::Redaction` — Null | Hash | Mask { keep_prefix, keep_suffix }
@@ -230,7 +230,7 @@
- Profile-driven routing: `POST /vectors/profile/{id}/search` auto-routes to Lance when profile.vector_backend=lance - Profile-driven routing: `POST /vectors/profile/{id}/search` auto-routes to Lance when profile.vector_backend=lance
- Auto-migrate + auto-index on activation - Auto-migrate + auto-index on activation
- Measured on real 100K × 768d: migrate 0.57s, IVF_PQ build 16.2s (14× faster than HNSW 230s), search 23ms, append 100 rows 3.3ms, doc_id fetch 3.5ms (with scalar btree) - Measured on real 100K × 768d: migrate 0.57s, IVF_PQ build 16.2s (14× faster than HNSW 230s), search 23ms, append 100 rows 3.3ms, doc_id fetch 3.5ms (with scalar btree)
- IVF_PQ recall@10 = 0.805 (HNSW = 1.000) — measured via `/vectors/lance/recall/{idx}` harness - IVF_PQ recall@10 = 0.805 with Lance's default `nprobes=1` (the hidden cap — see 2026-04-20 tuning work below, which lifts it to 1.000). Measured via `/vectors/lance/recall/{idx}` harness.
- [x] Phase E.3: Scheduled ingest — 2026-04-17 - [x] Phase E.3: Scheduled ingest — 2026-04-17
- `ingestd::schedule` module: ScheduleDef, ScheduleStore (JSON at `_schedules/{id}.json`), Scheduler tokio task - `ingestd::schedule` module: ScheduleDef, ScheduleStore (JSON at `_schedules/{id}.json`), Scheduler tokio task
- Supports MySQL + Postgres sources on interval triggers (Cron variant defined, parsing stubbed) - Supports MySQL + Postgres sources on interval triggers (Cron variant defined, parsing stubbed)
@@ -241,10 +241,93 @@
- Two-tier: lopdf text extraction → Tesseract 5.5 fallback for scanned/image PDFs - Two-tier: lopdf text extraction → Tesseract 5.5 fallback for scanned/image PDFs
- Extracts embedded XObject /Image streams, shells to tesseract --oem 3 --psm 6 - Extracts embedded XObject /Image streams, shells to tesseract --oem 3 --psm 6
- Same schema (source_file, page_number, text_content) — downstream unchanged - Same schema (source_file, page_number, text_content) — downstream unchanged
- [ ] Fine-tuned domain models - [x] Catalog hygiene — idempotent `register()` + dedupe + DELETE (2026-04-19, ADR-020)
- [ ] Multi-node query distribution - `catalogd::Registry::register` now gates on `(name, schema_fingerprint)`: same fp → reuse `DatasetId` and update objects in place; different fp → return error (409 Conflict on HTTP, `FAILED_PRECONDITION` on gRPC). First-time registration is unchanged.
- `POST /catalog/dedupe` one-shot operator endpoint collapses pre-existing duplicates; winner = non-null `row_count` first, newest `updated_at` second.
- `DELETE /catalog/datasets/by-name/{name}` removes the manifest from both in-memory registry and object storage (metadata-only — parquet files, vector indexes, tombstones are NOT cascade-deleted). Added to support test-harness cleanup; also plugs a real catalog hole where zombie entries from prior deletes would break DataFusion schema inference.
- Cleanup run on live catalog: 374 → 31 datasets, 343 orphan manifests removed, 0 errors. 308× `successful_playbooks` was the worst offender.
- Concurrency: write lock held across storage I/O in `register()` to close the check→insert TOCTOU window (32-worker multi-threaded stress test verifies single-manifest invariant).
- End-to-end verification: `scripts/e2e_pipeline_check.sh` runs 31 assertions across 12 pipeline stages (ingest → catalog → SQL+JOIN → dedup → idempotency → metadata → PII → vector embed → semantic search → cleanup) against the live gateway. Idempotent across repeat runs.
- Tests: 11 new in `catalogd` (was 0, includes 3 concurrency tests + 3 delete_dataset tests); 11 new in `storaged` for `AppendLog` + `ErrorJournal` (was 0). Fixed a broken doctest in `append_log.rs`.
- [x] Autotune agent: portfolio rotation + auto-bootstrap (2026-04-20)
- `pick_periodic_target` now sources candidates from `IndexRegistry` (not just promoted indexes) and picks least-recently-tuned, so trial budget spreads across every index with ≥1000 vectors instead of fixating on one converged champion.
- `run_one_cycle` bootstraps on first visit: `ensure_auto_harness` auto-generates `{index}_auto` (20 synthetic self-queries, k=10, brute-force ground truth) if missing, then seeds with `HnswConfig::default()` (ec=80/es=30).
- Regression fix: `harness::recall_at_k` now uses set-intersection semantics. The prior impl counted duplicates in `predicted` — on corpora with repeated chunks (`kb_response_cache_agent`) this inflated recall above 1.0 and poisoned promotion decisions. +7 unit tests.
- [x] Scheduled ingest: real cron parsing (2026-04-20)
- Vixie-compatible 5/6-field cron via `croner` crate. Day-of-week follows Unix convention (`1-5` = Mon-Fri). 6-field adds seconds granularity.
- `validate_trigger` in `ingestd::schedule` — create/patch handlers reject malformed expressions with `400 BAD_REQUEST` at creation time, not silently at fire time.
- Swapped away from the `cron` crate (0.16) which uses a non-Unix DOW convention (`1=Sun`) that would silently bite anyone writing `1-5` expecting weekdays. +9 unit tests.
- [x] EvalSets federation (2026-04-20)
- `harness::HarnessStore` mirrors the TrialJournal / PromotionRegistry federation pattern: eval artifacts colocate with each index's recorded bucket; legacy evals in primary remain discoverable via a fallback path; cross-bucket listing dedupes.
- Every eval callsite (service.rs × 5, agent.rs × 3, autotune.rs × 1) now routes through `HarnessStore`. `VectorState` and `AgentDeps` each hold a shared instance.
- [x] Index bucket-migrate PATCH (2026-04-20)
- `PATCH /vectors/indexes/{name}/bucket` copies an index's vector parquet + trial-journal batches + promotion file + auto-harness to `dest_bucket`, flips `IndexMeta.bucket` as the commit point, and evicts the `EmbeddingCache` so next load reads from the new bucket. Optional `delete_source: true` sweeps source artifacts.
- Lance-backed indexes refused with 400 — Lance URIs are bucket-specific and require rewriting the dataset, separate story. Round-trip verified: 390 artifacts, 0.04s.
- [x] IVF_PQ recall tuning (2026-04-20)
- `LanceVectorStore::search` now accepts optional `nprobes` + `refine_factor`. Lance's built-in `nprobes=1` default was the hidden cap on recall — on 316-partition `resumes_100k_v2` it searched only 0.3% of partitions per query.
- Server defaults (`LANCE_DEFAULT_NPROBES=20`, `LANCE_DEFAULT_REFINE_FACTOR=5`) flow through the scoped-search path and the autotune harness. Measured on `resumes_100k_v2`: recall `0.805 → 1.000` at p50 ≈ 7.4ms. Even `nprobes=5, refine=5` saturates recall at p50 ≈ 4.7ms.
- `/vectors/lance/recall/{idx}` accepts per-request `nprobes` / `refine_factor` so operators can sweep the curve.
- [x] **Phase 19: Playbook memory (meta-index)** — the feedback loop originally implied by the PRD but never built. Playbooks stop being write-only; they start shaping future rankings. (2026-04-20)
- [x] 19.1 — `POST /vectors/playbook_memory/rebuild` scans `successful_playbooks` via DataFusion, builds one `PlaybookEntry` per row (operation + approach + context embedded as one vector via nomic-embed-text)
- [x] 19.2 — Brute-force cosine search over in-memory embeddings (chosen over HNSW: successful_playbooks maxes around thousands of rows, overhead of a second indexed surface isn't worth it until that ceiling bites)
- [x] 19.3 — Endorsed names parsed out of `result` column, keyed by `(city, state, name)` tuple so shared names across cities don't cross-pollinate. Parsing via `parse_names` + `parse_city_state` helpers (7 unit tests)
- [x] 19.4 — `/vectors/hybrid?use_playbook_memory=true`: fetches `top_k * 5` candidates so endorsed workers outside the vanilla top-K can still climb. Boost is additive on vector score, each hit carries `playbook_boost` + `playbook_citations` in the response for explainability
- [x] 19.5 — Multi-agent orchestrator (`tests/multi-agent/orchestrator.ts`) auto-seeds `POST /vectors/playbook_memory/seed` on consensus_done, so the next query sees the new endorsement without a full `/rebuild`. Closes the feedback loop: two agents reach consensus → playbook sealed → next query re-ranks
- [x] 19.6 — `MAX_BOOST_PER_WORKER = 0.25` enforced in `compute_boost_for`; verified with unit test (100 identical playbooks → boost capped at 0.25) and live test (5 identical seeds → exactly 0.25). Time decay also wired: `BOOST_HALF_LIFE_DAYS = 30.0` — 30-day-old playbooks contribute half, 60-day a quarter, via `exp(-age_days / 30)` in the boost loop
- Real finding surfaced during build: the 32 bootstrap rows in `successful_playbooks` reference phantom worker names — 80 of 82 don't correspond to actual rows in `workers_500k`. `/seed` endpoint bypasses `successful_playbooks` so operators can prime memory with real fixtures; production path is the orchestrator write-through
- [x] **Phase 19 refinement — geo + role prefilter on boost** (2026-04-21)
- Added `compute_boost_for_filtered` and `compute_boost_for_filtered_with_role` to `playbook_memory.rs`. SQL filter's `(city, state, role)` parsed in `service.rs`; exact role-matches in target geo skip cosine and earn similarity=1.0. Restored the feedback loop: matched=0 → matched=11 per query on the same Nashville test. Citation density on Riverfront Steel: 2 → 28 per run (14×).
- Rust unit tests: `extractor_tests::extract_target_geo_basic/_missing_state/_word_boundary`, `extract_target_role_basic/_none/_multi_word`. 6/6 pass.
- Diagnostic log: `playbook_boost: boosts=N sources=N parsed=N matched=N target_geo=? target_role=?` on every call.
- [x] **Phase 20: Model Matrix + Overseer Tiers** (2026-04-21)
- `config/models.json` — 5 tiers (t1_hot / t2_review / t3_overview / t4_strategic / t5_gatekeeper), each with context_window + context_budget + overflow_policy. Ollama Cloud bearer key from `/root/llm_team_config.json`.
- Hot path: qwen3.5:latest + qwen3:latest local with `think:false`. Mistral dropped after 0/14 fill on complex scenarios.
- T3 cloud: gpt-oss:120b via Ollama Cloud — verified 4-8s latency, strict JSON-shape output for remediation.
- [x] **Phase 21: Scratchpad + Tree-Split Continuation** (2026-04-21)
- `tests/multi-agent/agent.ts`: `estimateTokens()`, `assertContextBudget()`, `generateContinuable()`, `generateTreeSplit()`. `think` flag plumbed through sidecar's `/generate`. Empty-response backoff + truncation-continuation, no max_tokens tourniquet.
- Rust port shipped (2026-04-21, companion to Phase 27):
- `crates/aibridge/src/context.rs` — `estimate_tokens` (chars/4 ceil, matches TS), `context_window_for`, `assert_context_budget` returning `Result<BudgetCheck, (BudgetCheck, usize over_by)>` so callers get the numbers back on both success and overflow. Windows table mirrors `config/models.json`.
- `crates/aibridge/src/continuation.rs` — `generate_continuable<G: TextGenerator>` handles the two failure modes from TS: (a) empty thinking-model response → geometric-backoff retry with 2× budget up to `budget_cap`; (b) truncated non-empty → continuation with partial-as-scratchpad. `is_structurally_complete` balances braces then JSON.parse-check for the JSON shape; text shape is "non-empty". Guards the degen case "all retries empty → bail, don't loop on empty partial" — the TS impl has this implicit, Rust makes it explicit.
- `crates/aibridge/src/tree_split.rs` — `generate_tree_split` map→reduce with running scratchpad. Per-shard + reduce-prompt budget checked through `assert_context_budget`; loud-fails with the overflow message rather than silently truncating. Scratchpad oldest-digest-first truncation once it exceeds `scratchpad_budget` (default 6000 tokens, matches TS).
- `TextGenerator` trait (native async-fn-in-trait, edition 2024) so `ScriptedGenerator` test double can inject canned sequences without a live Ollama. `AiClient` implements `TextGenerator`.
- `GenerateRequest` gained `think: Option<bool>` field — forwards to sidecar for per-call hidden-reasoning opt-out on hot-path JSON emitters.
- 25 aibridge unit tests (8 context + 10 continuation + 7 tree_split) — all green, no network required.
- [x] **Phase 22: Internal Knowledge Library** (2026-04-21)
- `data/_kb/` — signatures.jsonl, outcomes.jsonl, pathway_recommendations.jsonl, error_corrections.jsonl, config_snapshots.jsonl. Event-driven cycle: indexRun → recommendFor → loadRecommendation.
- Item B cloud rescue: failed event → cloud remediation JSON → retry with pivot. Verified 1/3 rescues succeeded on stress_01 (Gary IN → South Bend IN pivot).
- `scripts/kb_measure.py` aggregator. Unit tests: `kb.test.ts` — 4/4 pass (signature stability, role/city/count invariants, digest shape).
- [x] **Phase 23: Staffer identity + competence-weighted retrieval** (2026-04-21)
- ScenarioSpec gained `contract: ContractTerms` and `staffer: Staffer { id, name, tenure_months, role, tool_level }`.
- tool_level runtime overrides: full / local / basic / minimal. Basic + minimal route executor to Ollama Cloud `kimi-k2.5` (kimi-k2.6 pending pro-tier upgrade).
- `data/_kb/staffers.jsonl` — competence_score = 0.45·fill + 0.20·turn_eff + 0.20·cite + 0.15·rescue. Recomputed per run.
- `findNeighbors` now returns `weighted_score = cosine × max_staffer_competence`. `scripts/kb_staffer_report.py` — leaderboard + cross-staffer worker overlap (Rachel D. Lewis 12× across 4 staffers → auto-discovered high-value label).
- `gen_staffer_demo.ts` + `run_staffer_demo.sh` — 4 personas × 3 contracts = 12 runs.
- [x] **Phase 27: Playbook versioning** (2026-04-21)
- `PlaybookEntry` gained `version: u32` (default 1), `parent_id`, `superseded_at`, `superseded_by` fields. All `#[serde(default)]` so entries persisted before Phase 27 load as roots with version=1.
- `PlaybookMemory::revise_entry(parent_id, new_entry)` appends a new version, stamps `superseded_at`+`superseded_by` on the parent, inherits `parent_id` and sets `version = parent + 1` on the new entry. Rejects revising a retired or already-superseded parent with a clear error — the tip of the chain is the only valid revise target.
- `PlaybookMemory::history(playbook_id)` returns the full chain root→tip, walking `parent_id` backward then `superseded_by` forward. Cycle-safe. Works from any node in the chain.
- Superseded entries excluded from boost (same rule as retired): `compute_boost_for_filtered_with_role`, the active-entries prefilter, the geo-index rebuild, and the upsert existing-entry search all skip `superseded_at.is_some()`.
- Endpoints: `POST /vectors/playbook_memory/revise` + `GET /vectors/playbook_memory/history/{id}`.
- `status_counts` now returns `(total, retired, superseded, failures)`. `/status` JSON reports `superseded` as a distinct counter; `active = total - retired - superseded`.
- 8 unit tests under `mod version_tests` covering: chain-metadata stamping, retired-parent rejection, already-superseded-parent rejection, superseded endorsement exclusion from boost, history traversal from root/tip/middle, empty-for-unknown-id, superseded-status-count, legacy-entry-default-version round-trip. 26/26 playbook_memory tests pass.
- [x] **Phase 24: Observer / Autotune integration** (2026-04-20, commit b95dd86)
- Closed the gap where `lakehouse-observer.service` wrapped MCP :3700 while `tests/multi-agent/scenario.ts` hit gateway :3100 directly — observer sat idle at 0 ops across 3600+ cycles.
- `observer.ts` gained a Bun HTTP listener on `OBSERVER_PORT` (default 3800) with `GET /health`, `GET /stats` (totals + by_source + recent scenario digest), and `POST /event` for scenario outcomes. Body shapes into `ObservedOp` with `source="scenario"` + `staffer_id` + `sig_hash` + `event_kind` + geo + rescue flags.
- `recordExternalOp()` shared ring-buffer insert — ERROR_ANALYZER and PLAYBOOK_BUILDER loops now see both MCP-wrapped and scenario-posted ops through the same path.
- `persistOp()` swap: old path wrote via `/ingest/file?name=observed_operations` which has REPLACE semantics (wiped prior ops); now uses append-friendly Parquet write-through.
- [x] **Phase 25: Validity windows + playbook retirement** (2026-04-21, commit e0a843d)
- `PlaybookEntry` gained four optional fields (`#[serde(default)]` so legacy entries load as never-expiring): `schema_fingerprint` (SHA-256 over target dataset columns at seed time), `valid_until` (RFC3339 hard expiry), `retired_at` (set by retire calls), `retirement_reason` (human string).
- `compute_boost_for_filtered_with_role` now skips retired + expired entries before geo/cosine — no silent boosting from stale playbooks. Unit-tested on expired `valid_until` + retired + schema-drift retirement.
- Two retirement paths: `retire_one(playbook_id, reason)` for manual, `retire_on_schema_drift(city, state, current_fingerprint, reason)` for batch schema-migration sweep. Legacy entries without a fingerprint skip drift retirement (safe).
- Endpoint: `POST /vectors/playbook_memory/retire` — accepts either `{playbook_id, reason}` or `{city, state, current_schema_fingerprint, reason}`.
- [x] **Phase 26: Mem0 upsert + Letta geo hot cache** (2026-04-21, commit 640db8c)
- Mem0-style upsert: `/seed` with `append=true` (default) routes through `upsert_entry`, which decides ADD / UPDATE / NOOP on (operation, day, city, state). Same-day re-seed merges names (union, stable order) instead of duplicating the row. Identical re-seed is a no-op. Different-day same-operation is a fresh ADD. Playbook_id stays stable on UPDATE so prior citations remain valid.
- Letta-style hot cache: `PlaybookMemory` now holds a `geo_index: HashMap<(city_lower, state_upper), Vec<entry_idx>>` rebuilt on every mutation. Geo-filtered boost queries skip the full scan and hit the O(1) key lookup. At 1.9K entries the full scan was sub-ms; the index scales the same path to 100K+ without code changes.
- `UpsertOutcome` enum reported back to callers — `{mode: added|updated|noop, playbook_id, merged_names?}` + `entries_after`.
- [ ] Fine-tuned domain models (Phase 25+)
- [ ] Multi-node query distribution (only if ceilings bite)
--- ---
**52+ unit tests | 13 crates | 19 ADRs | 2.47M rows | 100K vectors | Hybrid Parquet+HNSW ⊕ Lance** **145 unit tests | 13 crates | 21 ADRs | 2.47M rows | 100K vectors | Hybrid Parquet+HNSW ⊕ Lance | Phases 0-27 shipped**
**Latest: 2026-04-17 — 8 commits shipping Phase 16.2 through Phase 18** **Latest: 2026-04-21 — Phase 27 (playbook versioning: `version` + `parent_id` + `superseded_at` + `superseded_by` on `PlaybookEntry`, `/revise` + `/history` endpoints, 8 new tests). Doc-sync pass: Phase 24 observer + Phase 25 validity windows + Phase 26 Mem0/Letta now reflected in phase tracker. Phase 19.6 time decay noted as wired (was misdocumented as deferred). Phase E.2 tombstone-at-compaction noted as closed in Phase 8 MVP limits.**
docs/PRD.md
+382 -4
View File
@@ -1,8 +1,28 @@
# PRD: Lakehouse — Rust-First Substrate for Versioned Knowledge Stores # PRD: Lakehouse — Rust-First Substrate for Versioned Knowledge Stores
**Status:** Active — Phases 0-18 shipped; hybrid SQL+Vector search operational; IVF_PQ recall tuned to 0.97; 10K real staffing profiles ingested from Ethereal; autonomous agent + staffing simulation verified > ## ▶ Direction pivot — 2026-04-22
>
> **`docs/CONTROL_PLANE_PRD.md` is now the long-horizon architecture target.**
> Lakehouse (this document) is being refactored to conform to the Universal AI
> Control Plane architecture — six layers, `/v1/*` universal API, multi-provider
> routing, Truth Layer for DevOps constraints.
>
> This PRD remains the source of truth for **everything Phases 0-37** — the
> shipped staffing / AI-substrate system below is preserved as the reference
> implementation and the first domain-specific consumer of the control plane.
> All existing phases, ADRs, and invariants continue to apply.
>
> **Phases 38+ live in `CONTROL_PLANE_PRD.md`** (Universal API, provider
> adapters, routing engine, expanded profile system, Truth Layer, validation
> pipeline, caller migration).
>
> Cross-read: this PRD for what's shipped, CONTROL_PLANE_PRD for where it's going.
---
**Status:** Active — Phases 0-36 shipped, Phase 37 (hot-swap async) TODO; control-plane pivot in flight; E2E test 9.0/10; stress test passes
**Created:** 2026-03-27 **Created:** 2026-03-27
**Last updated:** 2026-04-17 — hybrid search closes the SQL+vector gap; quality eval surfaced and fixed real bugs **Last updated:** 2026-04-22 — Control-plane pivot; Gateway seed concurrency (Semaphore); stress test with 6 diverse tasks
**Owner:** J **Owner:** J
--- ---
@@ -80,6 +100,9 @@ No new frameworks without documented ADR.
| **aibridge** | Rust↔Python boundary — HTTP client to FastAPI sidecar | | **aibridge** | Rust↔Python boundary — HTTP client to FastAPI sidecar |
| **ui** | Dioxus frontend — Ask, Explore, SQL, System tabs | | **ui** | Dioxus frontend — Ask, Explore, SQL, System tabs |
| **shared** | Types, errors, Arrow helpers, config, protobuf definitions, **secrets provider trait**, **PII detection** | | **shared** | Types, errors, Arrow helpers, config, protobuf definitions, **secrets provider trait**, **PII detection** |
| **mcp-server** | Agent gateway (Bun) — MCP tools, intelligence endpoints, scenario observer (:3700) |
| **observer** | Autonomous iteration loop — records ops, error analysis, playbook consolidation (:3800) |
| **scenario** | Day-in-the-life orchestrator — T5 tiers, KB integration, continuation tree-split (:bun) |
**Federation building blocks** (shipped 2026-04-16): **Federation building blocks** (shipped 2026-04-16):
- `shared::secrets::SecretsProvider` trait + `FileSecretsProvider` reading `/etc/lakehouse/secrets.toml` (0600 enforced) - `shared::secrets::SecretsProvider` trait + `FileSecretsProvider` reading `/etc/lakehouse/secrets.toml` (0600 enforced)
@@ -126,6 +149,15 @@ User question → gateway
9. **Every reader gets its own profile.** Human operators, AI agents, and local models are all clients of the same substrate. Each has a named profile with its own bucket, vector indexes, trial history, and dataset bindings. Profiles are a first-class architectural concept, not a tenancy afterthought. (Phase 17) 9. **Every reader gets its own profile.** Human operators, AI agents, and local models are all clients of the same substrate. Each has a named profile with its own bucket, vector indexes, trial history, and dataset bindings. Profiles are a first-class architectural concept, not a tenancy afterthought. (Phase 17)
10. **Trials are data, not logs.** Every index build is a trial with measurable metrics. The trial journal IS the agent's memory for how to tune itself. Stored as write-once batched JSONL per the ADR-018 append-log pattern. 10. **Trials are data, not logs.** Every index build is a trial with measurable metrics. The trial journal IS the agent's memory for how to tune itself. Stored as write-once batched JSONL per the ADR-018 append-log pattern.
11. **Operational failures are findable in one HTTP call.** The bucket error journal, trial journal, and audit log all expose `/storage/errors`, `/hnsw/trials`, `/access/audit` with structured filter + aggregation. No `grep` archaeology to answer "what broke?" 11. **Operational failures are findable in one HTTP call.** The bucket error journal, trial journal, and audit log all expose `/storage/errors`, `/hnsw/trials`, `/access/audit` with structured filter + aggregation. No `grep` archaeology to answer "what broke?"
12. **Playbooks feed the index, not just the log.** A completed playbook isn't just a record of what worked — it's a signal that shapes future rankings. Every `successful_playbooks` row contributes to the playbook-memory vector index, so semantically-similar future operations re-rank toward workers that have actually succeeded in comparable fills. This is the "system gets smarter over time" dimension that distinguishes this substrate from a static search engine. (Phase 19)
---
## Vision drift acknowledged (2026-04-20)
The system as shipped through Phase 18 is a **hybrid SQL+vector search engine with a playbook log**. The original pitch (and the "staffing AI co-pilot" framing) implied a **meta-index that learns from playbooks over time** — hot-swap profiles weren't just routing, they were knowledge generations that compounded. That learning loop was never built; playbooks were write-only. Phase 19 closes that gap explicitly.
The feedback signal is **statistical + semantic**, not neural. No model training — the index reads the playbook journal, computes operation-similarity, and boosts endorsed workers at query time. Rebuildable from `successful_playbooks` alone, same as every other derived index.
--- ---
@@ -349,11 +381,357 @@ The question raised 2026-04-16 after J's LLMS3 knowledge base identified Lance a
Per-profile `vector_backend: Parquet | Lance` becomes part of Phase 17 (model profiles). See ADR-019 for the full scorecard and caveats. Per-profile `vector_backend: Parquet | Lance` becomes part of Phase 17 (model profiles). See ADR-019 for the full scorecard and caveats.
### Phase 19+: Further horizon ### Phase 19: Playbook memory (meta-index) — the feedback loop
Make successful playbooks actually improve future searches. Today `successful_playbooks` is a write-only log; future-you looks at it and thinks "cool, we filled Toledo welders once" — but the index has no idea it happened, so the next Toledo-welder search ranks the same as if none of those fills had existed. Phase 19 closes the loop.
| Step | Deliverable | Gate |
|---|---|---|
| 19.1 | Embed every `successful_playbooks` row — operation + approach + context → one chunk per playbook | A new dataset `playbook_memory` appears in catalog with N rows = row count of `successful_playbooks` |
| 19.2 | Vector index on `playbook_memory` (HNSW or Lance — whichever `agent-parquet` profile uses) | `/vectors/search` against `playbook_memory` returns semantically similar past playbooks |
| 19.3 | Endorsement extraction: each playbook row has `fills[]` (worker_ids it succeeded with). Parse them out at ingest time and store in a sidecar `playbook_endorsements` Parquet keyed by playbook_id | `SELECT * FROM playbook_endorsements WHERE playbook_id = 'X'` returns the worker_ids |
| 19.4 | `/vectors/hybrid` gains opt-in `use_playbook_memory: bool`. When true: after hybrid ranks candidates, find top-K similar past playbooks (semantic search over `playbook_memory`), extract endorsed worker_ids, add a bounded boost to candidates in the endorsed set, re-rank | A search where the "right" worker is known from a prior playbook ranks higher with the flag than without |
| 19.5 | Write-through from multi-agent orchestrator: when two agents seal a playbook, it appends to `successful_playbooks` AND triggers a refresh of `playbook_memory` (via existing Phase C stale-mark path). Next query sees the new signal. | Run the orchestrator → inspect `playbook_memory` → see a new row. Run the same query before/after → ranking differs. |
| 19.6 | Ceiling-aware boost: cap the per-worker boost so one popular worker can't dominate future searches. Boost decays with time (optional) so stale playbooks matter less. | Synthetic test: 100 playbooks all filled with the same worker_id; the 101st search still returns a mix, not just that one worker |
**Gate:** Run a real search before and after a new successful playbook lands. The endorsed workers from similar past operations rank higher in the second call. Demonstrable with a diff of the two result sets.
**Why this is the right version of "meta-index":** The alternative — training a neural re-ranker on (query, candidate, outcome) triples — is a weeks-long ML story and requires labeled outcome data we don't really have. The statistical-semantic version here is rebuildable from the existing playbook journal, explainable ("boosted because of similar playbooks X, Y, Z"), and invalidatable (delete a playbook → boost goes away on next rebuild). It gets 80% of the payoff at 10% of the cost. Neural re-ranking stays as a future option if it bites.
**Non-goals for this phase:**
- Neural training / fine-tuning. Statistical feedback only.
- Hard guarantees about recall lift magnitude. "Measurably better on the demo query" is the gate, not a universal quality claim.
- Real-time recomputation on every playbook. Batched refresh via the existing stale-marking path is sufficient.
### Phase 19 refinement (WIRED 2026-04-21): geo-filter + role prefilter on boost
Item-3 diagnostic pass surfaced that `compute_boost_for` was ranking playbooks globally by cosine similarity, while candidates came from an SQL-filtered city. Result: boost map had 170 endorsed workers, 0 intersected the 50 Nashville-filtered candidates. Zero citations where there should have been dozens.
Fix — in `crates/vectord/src/playbook_memory.rs`:
- `compute_boost_for_filtered(target_geo)` — skip playbooks from other cities before cosine sort.
- `compute_boost_for_filtered_with_role(target_geo, target_role)` — multi-strategy: exact (role, city, state) match earns similarity=1.0 and fills up to half the top_k; cosine fallback fills the rest. Mirrors Mem0/Zep 2026 guidance on parallel-strategy rerank.
In `crates/vectord/src/service.rs`:
- `extract_target_geo` and `extract_target_role` pull both from the executor's SQL filter.
- `tracing::info!` emits `playbook_boost: boosts=N sources=N parsed=N matched=N target_geo=? target_role=?` on every hybrid_search. Silent-truncation class of bug now visible.
Citation lift measured: avg citations per run 0.32 → 1.38 after geo filter; then 2 → 28 in the single-scenario Riverfront Steel re-run after role prefilter landed. 14× delta on same scenario.
Unit tests: `extract_target_geo_basic`, `_missing_state_returns_none`, `_word_boundary` (rejects "civilian" substring), `extract_target_role_basic`, `_none_when_absent`, `_multi_word` — all pass (`cargo test -p vectord --lib extractor_tests`).
### Phase 20: Model Matrix + Overseer Tiers (WIRED 2026-04-21)
Five-tier routing declared in `config/models.json`. Hot path (T1/T2) stays local (qwen3.5 + qwen3 after mistral was dropped for 0/14 fill rate on complex scenarios). Cloud for overview (T3 gpt-oss:120b), strategic (T4 qwen3.5:397b), and gatekeeper (T5 kimi-k2-thinking). Every tier declares `context_window` + `context_budget` + `overflow_policy`.
- T1 hot: 50-200 calls/scenario, local only — `qwen3.5:latest` executor, `think:false`
- T2 review: 5-14 calls/event, local only — `qwen3:latest` reviewer, `think:false`
- T3 overview: 1-3 calls/scenario, cloud primary — `gpt-oss:120b` on Ollama Cloud, thinking on
- T4 strategic: 1-10 calls/day, cloud primary
- T5 gatekeeper: 1-5 calls/day, audit-logged
T3 checkpoints + cross-day lessons wired. Lessons archive to `data/_playbook_lessons/` and load back at next scenario start as `prior_lessons` in executor context. Cloud passthrough verified on stress_01 scenario with `LH_OVERVIEW_CLOUD=1` — `gpt-oss:120b` response latency consistently 4-8s, diagnosing city-pivot ("Gary IN → Chicago IL, 40mi") when target city has zero supply.
`think:false` is the key mechanical finding — qwen3.5 burns ~650 tokens of hidden reasoning before emitting response; hot-path JSON emitters MUST disable thinking or continuation has to paper over empty returns. T3/T4 overseers KEEP thinking (that's the point).
**Kimi-k2.6 upgrade path:** Current Ollama Cloud key returns 403 on kimi-k2.6 (`ollama run kimi-k2.6:cloud` requires `ollama signin` with pro-tier account). kimi-k2.5 substitutes on the current tier — same family, strong at tool calling. Swap to k2.6 is a one-line change in `applyToolLevel` once the subscription lands.
### Phase 21: Scratchpad + Tree-Split Continuation
**Why this is a phase and not an optimization:** bumping `max_tokens` until a response stops truncating is a tourniquet — J called this out explicitly. As playbooks accumulate into the hundreds and responses grow, eventually SOME request will exceed SOME model's window, and we can't solve it by raising a number. The stable answer is two primitives that let us handle arbitrary-size work without losing context: a scratchpad that glues multi-call responses together, and a tree split that shards oversized inputs and reduces them back.
**Two primitives (WIRED 2026-04-21 in `tests/multi-agent/agent.ts`):**
1. **`generateContinuable()`** — handles OUTPUT overflow. Calls the model; checks structural completeness (for JSON: matched braces + JSON.parse success; for text: non-empty). If incomplete, calls again with "continue from here" + the partial response as scratchpad. Up to `max_continuations` times. No `max_tokens` tuning needed — if thinking ate the initial budget, continuation picks up the slack.
2. **`generateTreeSplit()`** — handles INPUT overflow. Caller passes an array of shards (semantic chunks of the corpus). For each shard: map call with running scratchpad digest. Final reduce call produces the answer. Scratchpad truncates oldest content if it approaches its own budget. If a single shard still overflows, `assertContextBudget` throws — caller must re-shard at finer granularity, NOT silently truncate.
**Guarantees:**
1. No agent call can silently truncate. Either it completes, continues, or throws with numbers.
2. No corpus is too big — `generateTreeSplit` handles any size the caller can shard.
3. Scratchpad is the glue between multi-call responses; context is never lost, only compacted.
4. Token estimation uses `chars / 4` (biased safe ~15%) until we wire the provider's tokenizer.
**What lives where now:**
- `agent.ts::estimateTokens()` + `assertContextBudget()` + `generateContinuable()` + `generateTreeSplit()` — WIRED
- `scenario.ts` executor + reviewer + overviewGenerate calls — migrated to `generateContinuable`
- `config/models.json` — context_window + context_budget + overflow_policies per tier (declarative)
**Next sprint (Rust side, so gateway tools share it):**
- `crates/aibridge/src/continuation.rs` — port of `generateContinuable`
- `crates/aibridge/src/tree_split.rs` — port of `generateTreeSplit`
- `crates/storaged/src/chunk_cache.rs` — precomputed shards keyed by corpus hash (avoid re-chunking on every T4 run)
- `/metrics` counter: `context_continuations_total{model,shape,succeeded}`
**Status:** TS primitives WIRED. Rust port pending. The escalation path (tree split → bigger-context cloud model → kimi-k2:1t's 1M window → split decision into sub-decisions) is declared in `config/models.json` under `context_management.overflow_policies`.
### Phase 21 status update (WIRED 2026-04-21 evening)
Additional primitives landed after the initial commit:
- **`think: boolean`** flag plumbed through `generate()`, `generateCloud()`, `generateContinuable()`, and into sidecar's `/generate` endpoint. Enables per-call opt-out of hidden reasoning for hot-path JSON emitters. Verified: qwen3.5 with `think:false` + `num_predict:400` returns clean `{"worker_id":...}` on first call; without `think:false`, 650 tokens eaten by reasoning, response empty.
- **Cloud executor routing** — `ACTIVE_EXECUTOR_CLOUD` / `ACTIVE_REVIEWER_CLOUD` flags let per-staffer tool_level route executor to Ollama Cloud when weak local model (qwen2.5) would collapse. Verified on kimi-k2.5 via Ollama Cloud: clean JSON emission, think:false honored.
Rust port of continuation + tree-split primitives remains queued for next sprint (`crates/aibridge/src/continuation.rs`, `tree_split.rs`).
### Phase 22: Internal Knowledge Library (KB)
Meta-layer over Phase 19 playbook_memory. Playbook memory answers "which WORKERS worked for this event." The KB answers "which CONFIG worked for this playbook signature." Subject changes from workers to the system itself — model choice, budget hints, overflow policies, pathway notes.
**Files (`data/_kb/`):**
- `signatures.jsonl` — (sig_hash, embedding[], first_seen, last_seen, run_count). Sig = stable hash of the sequence of (kind, role, count, city, state) across events.
- `outcomes.jsonl` — per-run record: {sig, run_id, models, ok/total, turns, citations, per-event summary, elapsed}.
- `pathway_recommendations.jsonl` — AI-synthesized for next run: {confidence, rationale, top_models, budget_hints, pathway_notes, neighbors_consulted}.
- `error_corrections.jsonl` — detected fail→succeed pairs on same sig, diff of what changed.
- `config_snapshots.jsonl` — history of models.json changes + why.
**Cycle (event-driven, not wall-clock):**
1. Scenario ends → `kb.indexRun()` extracts signature, embeds spec digest, appends outcome.
2. `kb.recommendFor(nextSpec)` finds k-NN signatures via cosine, feeds their outcome history + recent error corrections to the overview model, writes a structured recommendation.
3. Next scenario starts → `kb.loadRecommendation(spec)` pulls the newest rec for this sig, injects `pathway_notes` into `guidanceFor()` alongside prior lessons.
**Why file-based for MVP:** Phase 19 playbook_memory is already a catalogd dataset. KB is a separate meta-layer; keep it file-based first to iterate without a gateway schema migration. Rust port (and promotion to vectord-indexed corpus for neighbor search at scale) lands once shape stabilizes — mirrors how Phase 21 primitives were TS-first → Rust next sprint.
**What the overview model gets asked:**
- Target scenario digest
- Top-k neighbor signatures with avg ok rate, best model combo per neighbor
- Recent error corrections (sig, before/after model set)
**What it outputs (JSON-constrained):**
- confidence (high/medium/low)
- rationale (2-3 sentences)
- top_models {executor, reviewer, overview}
- budget_hints {executor_max_tokens, reviewer_max_tokens, executor_think}
- pathway_notes (concrete pre-run advice)
**Status (WIRED 2026-04-21):** `tests/multi-agent/kb.ts` holds all primitives. scenario.ts reads rec at start, indexes + recommends at end. Cold start gracefully writes a "low confidence, no history" rec so the second run has a floor to build on. `snapshotConfig()` wired to fire at every scenario start — active model set + tool_level + cloud flags hashed and appended to `config_snapshots.jsonl`.
**Phase 22 item B — cloud rescue (WIRED):** When an event fails and cloud T3 is enabled, `requestCloudRemediation()` feeds the failure trace (SQL filters attempted, row counts, reviewer drift reasons, gap signals, contract terms) to cloud and parses a JSON remediation with new_city / new_state / new_role / new_count / rationale. Event retries once with the pivot. Verified 1/3 rescues succeeded on stress_01 (Gary IN → South Bend IN pivot filled a Welder that local drift-aborted). Sanitizer splits "City, ST" comma-packed outputs so downstream SQL doesn't get `Hammond, IN, IN`.
### Phase 23: Staffer identity + competence-weighted retrieval (WIRED 2026-04-21)
Answers "who handled this" as a first-class dimension of the matrix index. Senior staffers' playbooks rank higher than juniors' on similar scenarios via competence × similarity score. Auto-discovers "reliable performer" worker labels via cross-staffer endorsement overlap.
**Schema (`scenario.ts` ScenarioSpec):**
- `contract?: ContractTerms` — deadline, budget_per_hour_max, local_bonus_per_hour, local_bonus_radius_mi, fill_requirement. Propagates into T3 checkpoint + cloud rescue prompts so cloud reasons about trade-offs (pivot-within-radius before budget-pivot-further).
- `staffer?: Staffer` — {id, name, tenure_months, role, tool_level}. tool_level controls subsystems available to this run:
- `full` — qwen3.5 + qwen3 local + cloud T3 + cloud rescue
- `local` — qwen3.5 + qwen3 local + local gpt-oss:20b T3 + rescue
- `basic` — **kimi-k2.5 cloud** exec + qwen3 local reviewer + local T3, no rescue
- `minimal` — kimi-k2.5 cloud exec + qwen3 local reviewer, NO T3, NO rescue — tests whether playbook inheritance carries knowledge alone
**KB staffer indexing (`data/_kb/staffers.jsonl`):**
- Recomputed per-staffer on every run: total_runs, fill_rate, avg_turns_per_event, avg_citations_per_run, rescue_rate, competence_score.
- `competence_score = 0.45·fill_rate + 0.20·turn_efficiency + 0.20·citation_density + 0.15·rescue_rate`. Bounded 0..1.
**Weighted neighbor retrieval:**
- `findNeighbors` in `kb.ts` returns `weighted_score = cosine × max_staffer_competence` (floor 0.3). Senior playbooks rank above junior playbooks on similar scenarios.
- `pathway_recommendations` include `best_staffer_id` / `best_staffer_competence` so cloud knows WHOSE playbook it's synthesizing from.
**Cross-staffer auto-discovery:**
- `scripts/kb_staffer_report.py` emits leaderboard + workers endorsed across ≥2 staffers on same signature.
- Validated output: Rachel D. Lewis (Welder Nashville) endorsed 12× across 4 staffers; Christina Watson (Machine Op Indianapolis) 11×. These are the highest-confidence "reliable performer" labels the system produced without human tagging.
**Demo infrastructure:**
- `tests/multi-agent/gen_staffer_demo.ts` — 4 personas × 3 contracts = 12 scenario specs.
- `scripts/run_staffer_demo.sh` — sequential batch with cloud T3.
- `scripts/kb_staffer_report.py` — leaderboard + top/bottom differential + cross-staffer overlap.
### Phase 24: Observer / Autotune integration (SHIPPED 2026-04-20, commit b95dd86)
The gap: `lakehouse-observer.service` wrapped MCP :3700, while `tests/multi-agent/scenario.ts` hit gateway :3100 directly. Observer idle at 0 ops across 3600+ cycles — scenarios invisible to ERROR_ANALYZER and PLAYBOOK_BUILDER, autotune running blind to outcomes.
**What shipped:**
- `observer.ts` Bun HTTP listener on `OBSERVER_PORT` (default 3800): `GET /health`, `GET /stats` (totals, by_source, recent scenario digest), `POST /event` for scenario outcomes.
- `ObservedOp` carries provenance — `source="scenario" | "mcp"` + `staffer_id` + `sig_hash` + `event_kind` + geo + rescue flags.
- `recordExternalOp()` — shared ring-buffer insert; main analyzer + playbook builder no longer care where the op came from.
- `persistOp()` fix: old path POSTed to `/ingest/file?name=observed_operations` which has REPLACE semantics (wiped prior ops); now uses append-friendly write-through.
### Phase 25: Validity windows + playbook retirement (SHIPPED 2026-04-21, commit e0a843d)
Zep 2026-era finding: temporal validity is the single highest-value memory-hygiene primitive. `PlaybookEntry` gained `schema_fingerprint` / `valid_until` / `retired_at` / `retirement_reason`. `compute_boost_for_filtered_with_role` skips retired + expired before geo/cosine ranking. Two retirement paths: `retire_one(id, reason)` for manual, `retire_on_schema_drift(city, state, fp, reason)` for batch migration sweep. Endpoint: `POST /vectors/playbook_memory/retire`.
### Phase 26: Mem0 upsert + Letta geo hot cache (SHIPPED 2026-04-21, commit 640db8c)
Same-day re-seed no longer duplicates rows. `/seed` with `append=true` routes through `upsert_entry` which decides ADD / UPDATE / NOOP on `(operation, day, city, state)`. Playbook_id stays stable on UPDATE so existing citations remain valid. `PlaybookMemory.geo_index: HashMap<(city, state), Vec<idx>>` rebuilt on every mutation; geo-filtered boost queries skip the scan and hit O(1) lookup — sub-ms at current scale, same code path scales to 100K+ entries.
### Phase 27: Playbook versioning (SHIPPED 2026-04-21)
`PlaybookEntry` gained `version: u32` (default 1), `parent_id`, `superseded_at`, `superseded_by` — all `#[serde(default)]` so pre-Phase-27 state loads as roots. `revise_entry(parent_id, new_entry)` appends a new version, stamps the parent superseded, rejects revising a retired or already-superseded parent. `history(id)` returns the root→tip chain from any node. Superseded entries excluded from boost (same rule as retired). Endpoints: `POST /vectors/playbook_memory/revise`, `GET /vectors/playbook_memory/history/{id}`. `/status` reports `superseded` as a distinct counter. 8 new tests; 51/51 vectord lib tests green.
### Phase 28: Agent Gateway + MCP Server (SHIPPED 2026-04-21)
The agent gateway wraps the Rust substrate with an MCP-first interface (Model Context Protocol), enabling Claude Code, GPT agents, and internal scripts to interact with the lakehouse through named tools rather than raw HTTP. Built on Bun (not Node), serving as the "front door" for all AI consumers.
**Files:** `mcp-server/index.ts` (2241 lines)
| Tool | Purpose | Endpoint |
|---|---|---|
| `search_workers` | Hybrid SQL+vector (the core) | `POST /vectors/hybrid` |
| `query_sql` | Analytical SQL on any dataset | `POST /query/sql` |
| `match_contract` | Find workers for a job order | `POST /match` |
| `get_worker` | Single worker by ID | `POST /worker/:id` |
| `rag_question` | Full RAG pipeline | `POST /vectors/rag` |
| `log_success` | Record operation → playbook | `POST /log` |
| `log_failure` | Record failed fill (negative signal) | `POST /log_failure` |
| `get_playbooks` | Retrieve past successes | `GET /playbooks` |
| `swap_profile` | Hot-swap model+data context | `POST /profile/:id` |
| `vram_status` | GPU introspection | `GET /vram` |
**HTTP Routes (internal agent consumption):**
- `/search` — hybrid search with client blacklist + rate enrichment
- `/verify` — claim verification against golden dataset
- `/context` — self-orientation for agents
- `/clients/:client/blacklist` — per-client worker exclusion
- `/memory/query` — unified memory surface (Phase 24 refinement)
- `/system/summary` — truthful row counts via SQL
- `/models/matrix` — read `config/models.json`
### Phase 29: Intelligence Suite (SHIPPED 2026-04-21)
Market intelligence endpoints surfaced through the MCP server — real-world demand signals from public data, cross-referenced with worker bench for staffing gap analysis.
**Files:** `mcp-server/index.ts` continued
| Endpoint | Purpose | Data Source |
|---|---|---|
| `/intelligence/market` | Building permits → demand forecast | Chicago Socrata API |
| `/intelligence/staffing_forecast` | 30-day demand by role | Chicago permits + bench |
| `/intelligence/permit_contracts` | Permits + Phase 19 ranked candidates | Chicago permits + Workers 500K |
| `/intelligence/activity` | Activity feed + learned patterns | successful_playbooks |
| `/intelligence/brief` | Parallel analytics across 500K profiles | Workers 500K |
| `/intelligence/chat` | Natural language → routed queries | Hybrid + RAG |
**Market signals captured:**
- Largest permits by cost (top 50)
- Work type breakout (electrical, mechanical, masonry, plumbing)
- Cross-reference with IL bench supply/available/reliable
- Demand forecast using $150K/worker industry heuristic
### Phase 30: Observer + Autotune Integration (SHIPPED 2026-04-20, commit b95dd86)
The gap: `lakehouse-observer.service` wrapped MCP :3700, while `tests/multi-agent/scenario.ts` hit gateway :3100 directly. Observer idle at 0 ops across 3600+ cycles — scenarios invisible to ERROR_ANALYZER and PLAYBOOK_BUILDER, autotune running blind to outcomes.
**Files:** `mcp-server/observer.ts` (335 lines)
**What shipped:**
- `observer.ts` Bun HTTP listener on `OBSERVER_PORT` (default 3800): `GET /health`, `GET /stats` (totals, by_source, recent scenario digest), `POST /event` for scenario outcomes.
- `ObservedOp` carries provenance — `source="scenario" | "mcp"` + `staffer_id` + `sig_hash` + `event_kind` + geo + rescue flags.
- `recordExternalOp()` — shared ring-buffer insert; main analyzer + playbook builder no longer care where the op came from.
- `persistOp()` fix: old path POSTed to `/ingest/file?name=observed_operations` which has REPLACE semantics (wiped prior ops); now uses append-friendly write-through to `data/_observer/ops.jsonl`.
**HTTP API:**
- `GET /health` — ring size
- `GET /stats` — totals, by_source, recent scenario ops, rescue stats
- `POST /event` — record scenario outcomes with provenance
### Phase 31: Scenario Orchestrator with Tiers (SHIPPED 2026-04-21)
A day-in-the-life scenario test running six events against the live substrate. Exercises baseline_fill, recurring, expansion, emergency, misplacement. Routes through the same executor/reviewer loop as the single-task orchestrator with event-specific constraints.
**Files:** `tests/multi-agent/scenario.ts` (1772 lines)
**Model Matrix (configurable per-tier):**
| Tier | Purpose | Model | Context Budget |
|---|---|---|---|
| T1 hot | 50-200 calls/scenario | qwen3.5:latest | 8K, think:false |
| T2 review | 5-14 calls/event | qwen3:latest | 8K, think:false |
| T3 overview | 1-3 calls/scenario | gpt-oss:120b | 32K, thinking on |
| T4 strategic | 1-10 calls/day | gpt-oss:120b | 64K |
| T5 gatekeeper | 1-5 calls/day | kimi-k2.6 | 128K, audit-logged |
**Staffer tool_levels (Phase 23):**
- `full` — qwen3.5 + qwen3 local + cloud T3 + cloud rescue
- `local` — qwen3.5 + qwen3 local + local gpt-oss:20b T3 + rescue
- `basic` — kimi-k2.5 cloud exec + qwen3 local reviewer, no rescue
- `minimal` — kimi-k2.5 cloud exec + qwen3 local reviewer, NO T3, NO rescue
**Key features:**
- `generateContinuable()` — handles output overflow with continuation
- `generateTreeSplit()` — handles input overflow via sharding
- `think:false` flag — disables hidden reasoning for hot-path JSON emitters
- Cloud executor routing — `ACTIVE_EXECUTOR_CLOUD` / `ACTIVE_REVIEWER_CLOUD` flags
- Cloud rescue (Phase 22B) — `requestCloudRemediation()` on failure
- KB integration at start (load recommendation) and end (index + recommend)
### Phase 32: Knowledge Library + Staffer Indexing (SHIPPED 2026-04-21)
Meta-layer over Phase 19 playbook_memory. Tracks which configs worked for which playbook signatures.
**Files:** `tests/multi-agent/kb.ts` (600 lines)
**Files under `data/_kb/`:**
- `signatures.jsonl` — (sig_hash, embedding[], first_seen, last_seen, run_count)
- `outcomes.jsonl` — per-run: {sig, run_id, models, ok/total, turns, citations}
- `pathway_recommendations.jsonl` — AI-synthesized for next run
- `error_corrections.jsonl` — detected fail→succeed pairs
- `config_snapshots.jsonl` — history of model changes
- `staffers.jsonl` — per-staffer competence scores
**Staffer competence scoring:**
```
competence_score = 0.45·fill_rate + 0.20·turn_efficiency + 0.20·citation_density + 0.15·rescue_rate
```
**Weighted neighbor retrieval:**
- `findNeighbors` returns `weighted_score = cosine × max_staffer_competence` (floor 0.3)
- Senior playbooks rank above junior playbooks on similar scenarios
### Phase 33: Validity Windows + Playbook Retirement (SHIPPED 2026-04-21)
Zep 2026-era finding: temporal validity is the single highest-value memory-hygiene primitive.
**What shipped:**
- `PlaybookEntry` gained `schema_fingerprint` / `valid_until` / `retired_at` / `retirement_reason`
- `compute_boost_for_filtered_with_role` skips retired + expired before geo/cosine ranking
- Two retirement paths: `retire_one(id, reason)` manual, `retire_on_schema_drift(city, state, fp, reason)` batch
- Endpoint: `POST /vectors/playbook_memory/retire`
### Phase 34: Mem0 Upsert + Geo Hot Cache (SHIPPED 2026-04-21, commit 640db8c)
Same-day re-seed no longer duplicates rows.
**What shipped:**
- `/seed` with `append=true` routes through `upsert_entry` — decides ADD / UPDATE / NOOP on `(operation, day, city, state)`
- Playbook_id stays stable on UPDATE so existing citations remain valid
- `PlaybookMemory.geo_index: HashMap<(city, state), Vec<idx>>` rebuilt on every mutation
- O(1) geo-filtered boost queries — sub-MS at current scale
### Phase 36: Gateway Seed Concurrency Fix (SHIPPED 2026-04-22)
Problem: Concurrent `playbook_memory/seed` calls caused socket collisions with the Python AI sidecar. Two staffing coordinators hitting seed simultaneously → one socket error.
Solution:
- Added `embed_semaphore: Arc<Semaphore>` to `VectorState` (permits=1)
- Seed endpoint acquires permit before embedding, releases after
- Serializes embed calls to sidecar, eliminates collision
E2E test result: 8.0/10 avg (persist=0 is expected — same day+op = NOOP, not add)
### Phase 37: Hot-Swap Endpoint Hang (TODO)
Problem: `POST /vectors/profile/{id}/activate` hangs for 2-4 minutes because it does heavy work synchronously:
- Loads embeddings into memory
- Builds HNSW indexes
- Preloads Ollama model
- Auto-provisions buckets
Current workaround: Client uses 5-second timeout (test passes but doesn't verify hot-swap).
**Required fix**: Convert to async job pattern:
1. Endpoint accepts request, spawns background task, returns job_id immediately
2. Client polls `GET /jobs/{job_id}` for status
3. Progress events streamed via SSE during activation
4. Background task updates job status on completion
This is a known architectural debt item - not blocking since tests use timeout workaround.
### Phase 35+: Further horizon
- PDF OCR for scanned documents (Tesseract integration)
- Specialized fine-tuned models per domain (staffing matcher, resume parser) - Specialized fine-tuned models per domain (staffing matcher, resume parser)
- Video/audio transcript ingest + multimodal embeddings - Video/audio transcript ingest + multimodal embeddings
- Neural re-ranker over (query, candidate, outcome) triples — only if Phase 19's statistical feedback plateaus
- True distributed query (DataFusion multi-node) — only if single-machine ceilings bite - True distributed query (DataFusion multi-node) — only if single-machine ceilings bite
--- ---
mcp-server/alerts.html
+249
View File
@@ -0,0 +1,249 @@
<!DOCTYPE html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>Lakehouse — Alerts</title>
<style>
*{margin:0;padding:0;box-sizing:border-box}
body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;color:#b0b8c4;font-size:14px;line-height:1.55;-webkit-font-smoothing:antialiased}
a{color:#58a6ff;text-decoration:none}
a:hover{color:#79c0ff}
.bar{background:#0d1117;padding:0 24px;height:56px;border-bottom:1px solid #171d27;display:flex;justify-content:space-between;align-items:center;position:sticky;top:0;z-index:10}
.bar h1{font-size:14px;font-weight:600;color:#e6edf3;letter-spacing:-0.2px}
.bar nav{display:flex;gap:2px}
.bar nav a{font-size:12px;color:#545d68;padding:6px 14px;border-radius:6px;transition:all 0.15s}
.bar nav a:hover{color:#e6edf3;background:#161b22}
.bar nav a.active{color:#e6edf3;background:#1c2333}
.wrap{max-width:900px;margin:0 auto;padding:28px 20px 60px}
h2{color:#e6edf3;font-size:20px;font-weight:700;letter-spacing:-0.3px;margin-bottom:4px}
.lede{color:#8b949e;font-size:13px;margin-bottom:18px;line-height:1.6;max-width:680px}
.card{background:#0d1117;border:1px solid #171d27;border-radius:12px;padding:18px;margin:10px 0}
.row{display:flex;justify-content:space-between;align-items:center;gap:12px;padding:10px 0;border-bottom:1px solid #171d27}
.row:last-child{border-bottom:none}
.row .k{color:#8b949e;font-size:12px}
.row .v{color:#e6edf3;font-weight:500;font-size:13px;font-family:ui-monospace,Menlo,monospace}
label{display:block;color:#8b949e;font-size:12px;margin-bottom:6px;margin-top:12px}
label:first-child{margin-top:0}
input[type=text],input[type=number],input[type=url]{width:100%;padding:10px 14px;background:#161b22;border:1px solid #21262d;border-radius:8px;color:#e6edf3;font-size:13px;outline:none;font-family:ui-monospace,Menlo,monospace}
input:focus{border-color:#388bfd}
.switch{display:flex;align-items:center;gap:8px;margin-top:4px}
.switch input{width:auto;accent-color:#1f6feb}
.switch span{color:#c9d1d9;font-size:13px}
.btn{padding:10px 18px;background:#1f6feb;border:none;border-radius:8px;color:#fff;font-size:13px;font-weight:600;cursor:pointer;margin-right:8px;margin-top:14px}
.btn:hover{background:#388bfd}
.btn:disabled{opacity:0.4;cursor:wait}
.btn.ghost{background:transparent;border:1px solid #21262d;color:#c9d1d9}
.btn.ghost:hover{background:#161b22}
.btn.green{background:#2ea043}
.btn.green:hover{background:#3fb950}
.entry{background:#161b22;border-radius:6px;padding:10px 14px;margin-bottom:6px;font-size:12px;font-family:ui-monospace,Menlo,monospace;white-space:pre-wrap;color:#c9d1d9;line-height:1.6}
.entry .at{color:#545d68;font-size:10px;display:block;margin-bottom:4px}
.tip{color:#8b949e;font-size:12px;line-height:1.7;padding:10px 14px;border-left:2px solid #21262d;margin:10px 0}
.tip strong{color:#c9d1d9}
.tip code{background:#161b22;padding:1px 5px;border-radius:3px;font-family:ui-monospace,Menlo,monospace;font-size:11px;color:#79c0ff}
.result{padding:14px;border-radius:8px;margin-top:10px;font-size:12px;line-height:1.6}
.result.ok{background:#0d2818;border:1px solid #2ea04360;color:#86efac}
.result.err{background:#3a1a1a;border:1px solid #f8514960;color:#fca5a5}
.result.dim{background:#161b22;border:1px solid #21262d;color:#8b949e}
.chan{display:inline-block;padding:3px 10px;border-radius:10px;font-size:10px;font-weight:600;margin-right:6px;letter-spacing:0.3px}
.chan.console{background:#1a2744;color:#79c0ff}
.chan.file{background:#1a3a2a;color:#3fb950}
.chan.webhook{background:#2a1a3a;color:#bc8cff}
.footer{border-top:1px solid #171d27;padding:20px;text-align:center;color:#3d444d;font-size:11px}
@media(max-width:720px){
.wrap{padding:20px 12px 40px}
.bar nav{display:none}
}
</style></head>
<body>
<div class="bar">
<h1>Lakehouse — Alerts</h1>
<nav>
<a href=".">Dashboard</a>
<a href="console">Walkthrough</a>
<a href="proof">Architecture</a>
<a href="spec">Spec</a>
<a href="onboard">Onboard</a>
<a href="alerts" class="active">Alerts</a>
<a href="workspaces">Workspaces</a>
</nav>
</div>
<div class="wrap">
<h2>Push alerts — make the system find you</h2>
<div class="lede">The daemon runs in the background and dispatches a concise digest whenever something worth notifying changes: a role escalates to tight or critical, a new staffing deadline falls within your warn window, or the playbook memory compounds meaningfully. Phone-first shops don't open dashboards — they need the system to reach out.</div>
<div class="card">
<h3 style="color:#e6edf3;font-size:15px;margin-bottom:10px">Current status</h3>
<div id="status"><div style="color:#8b949e">Loading…</div></div>
</div>
<div class="card">
<h3 style="color:#e6edf3;font-size:15px;margin-bottom:10px">Configuration</h3>
<form id="cfg-form">
<label class="switch"><input type="checkbox" id="enabled"> <span>Daemon enabled — fire digests on the interval</span></label>
<label>Check interval (minutes)</label>
<input type="number" id="interval" min="1" max="1440" value="15">
<label>Deadline warn window (days)</label>
<input type="number" id="deadline_warn_days" min="1" max="90" value="7">
<label>Webhook URL (Slack, Discord, or any POST-accepting endpoint — optional)</label>
<input type="url" id="webhook_url" placeholder="https://hooks.slack.com/services/... or https://discord.com/api/webhooks/...">
<label>Webhook label (your name for this channel)</label>
<input type="text" id="webhook_label" placeholder="e.g. #staffing-ops">
<div>
<button type="button" class="btn" id="save-btn">Save</button>
<button type="button" class="btn green" id="fire-btn">Fire a test digest now</button>
</div>
<div id="save-result"></div>
</form>
<div class="tip">
<strong>Webhook behavior.</strong> The daemon POSTs JSON <code>{text, digest}</code> to your webhook URL.
Slack and Discord both accept this shape if the URL is an incoming-webhook. The <code>text</code>
field is pre-formatted human-readable; the <code>digest</code> field is structured for a bot to parse.
Interval changes take effect on next server restart.
</div>
</div>
<div class="card">
<h3 style="color:#e6edf3;font-size:15px;margin-bottom:10px">Recent digests</h3>
<div id="recent"><div style="color:#8b949e">Loading…</div></div>
</div>
</div>
<div class="footer">Lakehouse · alerts · daemon log: <code>mcp-server/data/notifications.jsonl</code></div>
<script>
var P=location.pathname.indexOf('/lakehouse')>=0?'/lakehouse':'';
var A=location.origin+P;
function el(t,c,x){var e=document.createElement(t);if(c)e.className=c;if(x!==undefined)e.textContent=String(x);return e}
function apiGet(p){return fetch(A+p).then(function(r){return r.json()})}
function apiPost(p,b){return fetch(A+p,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(b||{})}).then(function(r){return r.json()})}
function loadStatus(){
apiGet('/alerts/config').then(function(r){
var host=document.getElementById('status');host.textContent='';
var cfg=r.config||{};
var st=r.state||{};
var rows=[
['Daemon',cfg.enabled?'enabled':'disabled'],
['Interval',(cfg.interval_minutes||'?')+' minutes'],
['Deadline warn window',(cfg.deadline_warn_days||7)+' days'],
['Webhook',cfg.webhook_url?(cfg.webhook_label||'configured'):'none'],
['Last run',st.last_run_at?new Date(st.last_run_at).toLocaleString():'never'],
];
rows.forEach(function(r){
var row=el('div','row');
row.appendChild(el('span','k',r[0]));
row.appendChild(el('span','v',r[1]));
host.appendChild(row);
});
// Populate form
document.getElementById('enabled').checked=!!cfg.enabled;
document.getElementById('interval').value=cfg.interval_minutes||15;
document.getElementById('deadline_warn_days').value=cfg.deadline_warn_days||7;
document.getElementById('webhook_url').value=cfg.webhook_url||'';
document.getElementById('webhook_label').value=cfg.webhook_label||'';
}).catch(function(e){
var host=document.getElementById('status');host.textContent='';
host.appendChild(el('div',null,'status unavailable: '+(e.message||e)));
});
}
function loadRecent(){
apiGet('/alerts/recent').then(function(r){
var host=document.getElementById('recent');host.textContent='';
var entries=r.entries||[];
if(entries.length===0){
host.appendChild(el('div',null,'No digests yet. The daemon fires every '+(document.getElementById('interval').value||15)+' minutes and only dispatches when something changes. Click "Fire a test digest now" to force one.'));
host.firstChild.style.color='#8b949e';
return;
}
entries.forEach(function(e){
var card=el('div','entry');
if(e.at) card.appendChild(el('span','at',new Date(e.at).toLocaleString()));
card.appendChild(document.createTextNode(e.text||JSON.stringify(e.digest)));
host.appendChild(card);
});
}).catch(function(e){
var host=document.getElementById('recent');host.textContent='';
host.appendChild(el('div',null,'recent digests unavailable: '+(e.message||e)));
});
}
function save(){
var btn=document.getElementById('save-btn');
btn.disabled=true;
var body={
enabled:document.getElementById('enabled').checked,
interval_minutes:Number(document.getElementById('interval').value)||15,
deadline_warn_days:Number(document.getElementById('deadline_warn_days').value)||7,
webhook_url:document.getElementById('webhook_url').value.trim(),
webhook_label:document.getElementById('webhook_label').value.trim(),
};
apiPost('/alerts/config',body).then(function(r){
btn.disabled=false;
var out=document.getElementById('save-result');out.textContent='';
if(r.saved){
var ok=el('div','result ok',r.note||'saved.');out.appendChild(ok);
loadStatus();
} else {
out.appendChild(el('div','result err',JSON.stringify(r)));
}
}).catch(function(e){
btn.disabled=false;
var out=document.getElementById('save-result');out.textContent='';
out.appendChild(el('div','result err','save failed: '+(e.message||e)));
});
}
function fire(){
var btn=document.getElementById('fire-btn');
btn.disabled=true;btn.textContent='Firing…';
apiPost('/alerts/fire',{}).then(function(r){
btn.disabled=false;btn.textContent='Fire a test digest now';
var out=document.getElementById('save-result');out.textContent='';
if(!r.fired){
out.appendChild(el('div','result dim',r.reason||'nothing to dispatch'));
} else {
var ok=el('div','result ok');
ok.appendChild(document.createTextNode('Dispatched to: '));
(r.channels||[]).forEach(function(c){
var chip=el('span','chan '+c,c);ok.appendChild(chip);
});
if(r.errors&&r.errors.length){
ok.appendChild(document.createElement('br'));
ok.appendChild(document.createTextNode('errors: '+r.errors.join(' | ')));
}
out.appendChild(ok);
setTimeout(loadRecent,500);
}
}).catch(function(e){
btn.disabled=false;btn.textContent='Fire a test digest now';
var out=document.getElementById('save-result');out.textContent='';
out.appendChild(el('div','result err','fire failed: '+(e.message||e)));
});
}
document.addEventListener('DOMContentLoaded',function(){
loadStatus();loadRecent();
document.getElementById('save-btn').addEventListener('click',save);
document.getElementById('fire-btn').addEventListener('click',fire);
});
</script>
</body></html>
mcp-server/console.html
+429 -378
View File
@@ -1,419 +1,470 @@
<!DOCTYPE html> <!DOCTYPE html>
<html><head> <html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"> <meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>Staffing Intelligence Console</title> <title>Lakehouse — What Your Staffing System Would Do</title>
<style> <style>
*{margin:0;padding:0;box-sizing:border-box} *{margin:0;padding:0;box-sizing:border-box}
body{font-family:-apple-system,system-ui,sans-serif;background:#06090f;color:#c9d1d9;font-size:14px;line-height:1.5} body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;color:#b0b8c4;font-size:14px;line-height:1.55;-webkit-font-smoothing:antialiased}
.bar{background:#0d1117;padding:14px 20px;border-bottom:1px solid #1b2130;display:flex;justify-content:space-between;align-items:center;flex-wrap:wrap;gap:8px} a{color:#58a6ff;text-decoration:none}
.bar h1{font-size:15px;font-weight:600;color:#f0f6fc;letter-spacing:0.5px} a:hover{color:#79c0ff}
.bar .rt{font-size:11px;color:#484f58}
.wrap{max-width:960px;margin:0 auto;padding:20px 16px} .bar{background:#0d1117;padding:0 24px;height:56px;border-bottom:1px solid #171d27;display:flex;justify-content:space-between;align-items:center;position:sticky;top:0;z-index:10}
.brief-grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(280px,1fr));gap:12px;margin-bottom:24px} .bar h1{font-size:14px;font-weight:600;color:#e6edf3;letter-spacing:-0.2px}
.bcard{background:#0d1117;border:1px solid #1b2130;border-radius:10px;padding:18px;position:relative;overflow:hidden} .bar nav{display:flex;gap:2px}
.bcard .label{font-size:9px;text-transform:uppercase;letter-spacing:1.5px;color:#484f58;margin-bottom:8px} .bar nav a{font-size:12px;color:#545d68;padding:6px 14px;border-radius:6px;transition:all 0.15s}
.bcard .headline{font-size:16px;font-weight:700;color:#f0f6fc;margin-bottom:4px} .bar nav a:hover{color:#e6edf3;background:#161b22}
.bcard .sub{font-size:12px;color:#8b949e;margin-bottom:12px} .bar nav a.active{color:#e6edf3;background:#1c2333}
.bcard .big{font-size:36px;font-weight:800;line-height:1} .bar .rt{font-size:11px;color:#545d68}
.bcard.pulse{border-left:3px solid #58a6ff}
.bcard.bench{border-left:3px solid #d29922} .wrap{max-width:1040px;margin:0 auto;padding:28px 20px 60px}
.bcard.gems{border-left:3px solid #3fb950}
.bcard.risk{border-left:3px solid #f85149} .chapter{margin-bottom:48px}
.bcard.talent{border-left:3px solid #bc8cff} .chapter .num{color:#545d68;font-size:11px;font-weight:600;letter-spacing:1.6px;text-transform:uppercase;margin-bottom:6px}
.bcard.supply{border-left:3px solid #79c0ff} .chapter h2{color:#e6edf3;font-size:24px;font-weight:700;letter-spacing:-0.4px;margin-bottom:8px;line-height:1.2}
.arch-row{display:flex;gap:4px;flex-wrap:wrap;margin-top:10px} .chapter .lede{color:#8b949e;font-size:14px;margin-bottom:18px;max-width:680px;line-height:1.6}
.arch-pill{padding:3px 10px;border-radius:10px;font-size:11px;font-weight:500}
.bench-row{display:flex;align-items:center;gap:8px;margin-bottom:6px;font-size:12px} .card{background:#0d1117;border:1px solid #171d27;border-radius:12px;padding:20px;margin-bottom:12px}
.bench-row .st{width:24px;font-weight:600;color:#f0f6fc} .grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(240px,1fr));gap:10px}
.bench-bar{flex:1;height:14px;background:#161b22;border-radius:4px;overflow:hidden;position:relative}
.bench-fill{height:100%;border-radius:4px;transition:width 0.8s} .stat-lg{padding:18px 20px}
.bench-row .pct{width:40px;text-align:right;font-size:11px;font-weight:600} .stat-lg .n{font-size:30px;font-weight:800;color:#e6edf3;letter-spacing:-1px;line-height:1}
.wk{display:flex;align-items:center;gap:10px;padding:8px;background:#161b22;border-radius:6px;margin-bottom:4px} .stat-lg .l{font-size:10px;color:#545d68;text-transform:uppercase;letter-spacing:1.2px;margin-top:8px;font-weight:600}
.wk .av{width:32px;height:32px;border-radius:8px;display:flex;align-items:center;justify-content:center;font-weight:700;font-size:12px;color:#f0f6fc;flex-shrink:0} .stat-lg .sub{font-size:12px;color:#8b949e;margin-top:4px}
.wk .info{flex:1;min-width:0}
.wk .nm{font-weight:600;color:#f0f6fc;font-size:13px} .pill{display:inline-block;padding:3px 10px;border-radius:10px;font-size:10px;font-weight:600;letter-spacing:0.3px;margin-right:6px}
.wk .det{color:#8b949e;font-size:11px;white-space:nowrap;overflow:hidden;text-overflow:ellipsis} .pill.real{background:#0d2818;color:#3fb950;border:1px solid #2ea04380}
.wk .badge{padding:2px 8px;border-radius:8px;font-size:10px;font-weight:600;flex-shrink:0} .pill.synth{background:#1a1a2e;color:#bc8cff;border:1px solid #5530988c}
.prompts{display:flex;gap:8px;flex-wrap:wrap;margin-bottom:16px} .pill.flow{background:#0d2340;color:#58a6ff;border:1px solid #1f6feb80}
.prompt-btn{padding:8px 16px;background:#161b22;border:1px solid #1b2130;border-radius:20px;color:#8b949e;font-size:12px;cursor:pointer;transition:all 0.2s}
.prompt-btn:hover{border-color:#58a6ff;color:#58a6ff;background:#0d1525} .row{display:flex;justify-content:space-between;align-items:center;gap:12px;padding:10px 14px;background:#0d1117;border:1px solid #171d27;border-radius:8px;margin-bottom:6px;font-size:13px}
.chat-box{background:#0d1117;border:1px solid #1b2130;border-radius:12px;overflow:hidden;margin-bottom:16px} .row:hover{border-color:#21262d}
.chat-messages{min-height:60px;max-height:600px;overflow-y:auto;padding:16px} .row .title{color:#e6edf3;font-weight:500}
.msg{margin-bottom:16px;animation:fadeIn 0.3s ease} .row .meta{color:#8b949e;font-size:11px;margin-top:2px}
@keyframes fadeIn{from{opacity:0;transform:translateY(8px)}to{opacity:1;transform:translateY(0)}} .row .val{color:#58a6ff;font-weight:600;white-space:nowrap}
.msg.user{text-align:right}
.msg.user .bubble{display:inline-block;background:#1f3d68;color:#c9d6ff;padding:10px 16px;border-radius:16px 16px 4px 16px;max-width:70%;text-align:left;font-size:13px} details{background:#0d1117;border:1px solid #171d27;border-radius:8px;padding:10px 14px;margin-bottom:6px}
.msg.system .bubble{background:#161b22;border:1px solid #1b2130;border-radius:4px 16px 16px 16px;padding:16px;font-size:13px} details summary{cursor:pointer;font-size:13px;color:#e6edf3;font-weight:500;list-style:none}
.msg .meta{font-size:10px;color:#484f58;margin-top:6px} details summary::-webkit-details-marker{display:none}
.msg .steps{margin-bottom:10px} details .body{padding-top:10px;font-size:12px;color:#8b949e}
.msg .step{font-size:11px;color:#58a6ff;padding:2px 0}
.msg .summary{font-size:14px;font-weight:600;color:#f0f6fc;margin-bottom:8px} .accent-l{border-left:3px solid #2ea043}
.msg .note{font-size:11px;color:#8b949e;font-style:italic;margin-top:8px;padding:8px;background:#0d1117;border-radius:6px} .accent-b{border-left:3px solid #1f6feb}
.result-card{background:#0d1117;border:1px solid #1b2130;border-radius:8px;padding:12px;margin-bottom:6px} .accent-a{border-left:3px solid #bc8cff}
.result-card .rc-name{font-weight:600;color:#f0f6fc;font-size:13px} .accent-w{border-left:3px solid #d29922}
.result-card .rc-detail{font-size:11px;color:#8b949e;margin-top:2px}
.result-card .rc-score{font-size:10px;color:#58a6ff;margin-top:4px} .worker{display:flex;align-items:center;gap:10px;padding:8px 10px;background:#161b22;border-radius:6px;margin-bottom:4px;font-size:12px}
.result-card .rc-skills{display:flex;gap:4px;flex-wrap:wrap;margin-top:6px} .worker .av{width:28px;height:28px;border-radius:6px;background:#1a2744;display:flex;align-items:center;justify-content:center;font-weight:600;color:#e6edf3;font-size:10px;flex-shrink:0}
.result-card .rc-skill{padding:2px 8px;background:#1a2744;color:#58a6ff;border-radius:8px;font-size:10px} .worker .info{flex:1;min-width:0}
.result-card .rc-cert{padding:2px 8px;background:#1a3a2a;color:#3fb950;border-radius:8px;font-size:10px} .worker .nm{color:#e6edf3;font-weight:500}
.source-card{background:#1a1a00;border:1px solid #854d0e;border-radius:8px;padding:12px;margin-bottom:10px} .worker .why{color:#545d68;font-size:11px;margin-top:1px}
.whatif-severity{display:inline-block;padding:3px 10px;border-radius:8px;font-size:11px;font-weight:700;margin-left:8px} .worker .score{color:#58a6ff;font-size:11px;font-weight:600;white-space:nowrap}
.input-bar{display:flex;gap:8px;padding:12px 16px;border-top:1px solid #1b2130;background:#0d1117}
.input-bar input{flex:1;padding:12px 16px;background:#06090f;border:1px solid #1b2130;border-radius:10px;color:#f0f6fc;font-size:14px;outline:none} .mem-chip{background:#0d2818;border:1px solid #2ea04360;border-radius:6px;padding:8px 12px;font-size:11px;color:#86efac;line-height:1.5;margin-top:6px}
.input-bar input:focus{border-color:#58a6ff} .mem-chip .l{color:#3fb950;font-weight:600;margin-right:6px}
.input-bar button{padding:12px 20px;background:#58a6ff;border:none;border-radius:10px;color:#0d1117;font-weight:700;font-size:14px;cursor:pointer}
.input-bar button:hover{background:#79c0ff} .boost-chip{display:inline-block;margin-left:6px;padding:2px 7px;border-radius:9px;font-size:9px;font-weight:600;background:#0d2818;border:1px solid #2ea043;color:#3fb950;vertical-align:middle}
.input-bar button:disabled{opacity:0.4;cursor:default}
.thinking{display:flex;align-items:center;gap:8px;color:#58a6ff;font-size:12px;padding:12px} .try-box{background:#0d1117;border:1px solid #171d27;border-radius:12px;padding:16px}
.dots span{animation:blink 1.4s infinite both;margin:0 1px} .try-box input{width:100%;padding:12px 16px;background:#161b22;border:1px solid #21262d;border-radius:8px;color:#e6edf3;font-size:13px;outline:none;margin-bottom:10px}
.dots span:nth-child(2){animation-delay:0.2s} .try-box input:focus{border-color:#388bfd}
.dots span:nth-child(3){animation-delay:0.4s} .try-box button{padding:10px 20px;background:#1f6feb;border:none;border-radius:8px;color:#fff;font-size:13px;font-weight:600;cursor:pointer}
@keyframes blink{0%,80%,100%{opacity:0.2}40%{opacity:1}} .try-box button:hover{background:#388bfd}
.supply-row{display:flex;justify-content:space-between;align-items:center;padding:5px 8px;font-size:12px;border-bottom:1px solid #0d1117} .try-box button:disabled{opacity:0.5;cursor:wait}
.supply-row:last-child{border-bottom:none}
.supply-role{color:#f0f6fc;font-weight:500} .footer{border-top:1px solid #171d27;padding:20px;text-align:center;color:#3d444d;font-size:11px}
.supply-nums{color:#8b949e;font-size:11px}
.ft{text-align:center;padding:16px;color:#2d333b;font-size:10px} .loading{color:#484f58;font-style:italic;padding:20px 0;text-align:center}
@media(max-width:768px){.brief-grid{grid-template-columns:1fr}.msg.user .bubble{max-width:90%}} .err{color:#f85149;font-size:12px;padding:10px}
</style></head><body>
.narr{color:#8b949e;font-size:13px;line-height:1.7;margin:10px 0;padding:10px 14px;border-left:2px solid #21262d}
.narr strong{color:#c9d1d9;font-weight:600}
.step-label{color:#58a6ff;font-size:12px;margin-top:12px;font-weight:600}
.step-body{color:#c9d1d9;font-size:13px;margin-top:2px}
@media(max-width:720px){
.wrap{padding:20px 12px 40px}
.chapter h2{font-size:20px}
.bar nav{display:none}
}
</style></head>
<body>
<div class="bar"> <div class="bar">
<h1>Staffing Intelligence Console</h1> <h1>Lakehouse — What Your Staffing System Would Do</h1>
<div class="rt" id="brief-time">Analyzing 500,000 profiles...</div> <nav>
<a href=".">Dashboard</a>
<a href="console" class="active">Walkthrough</a>
<a href="proof">Architecture</a>
<a href="spec">Spec</a>
<a href="onboard">Onboard</a>
<a href="alerts">Alerts</a>
<a href="workspaces">Workspaces</a>
</nav>
<div class="rt" id="hdr-time">Reading live state…</div>
</div> </div>
<div class="wrap"> <div class="wrap">
<div id="brief" class="brief-grid"><div style="grid-column:1/-1;text-align:center;padding:40px;color:#484f58">Scanning your entire workforce...</div></div>
<div class="prompts" id="prompts"></div> <div class="chapter">
<div class="chat-box"> <div class="num">Chapter 1</div>
<div class="chat-messages" id="chat"></div> <h2>Right now, this system is already thinking</h2>
<div class="input-bar"> <div class="lede">Before you touched anything, it pulled real Chicago building-permit data, measured demand, checked your bench, and began flagging roles that need attention. This isn't theoretical — open your browser network tab and watch the fetches land.</div>
<input type="text" id="q" placeholder="Ask anything about your workforce..." onkeydown="if(event.key==='Enter'&&!event.shiftKey)send()"> <div class="grid" id="ch1-stats"><div class="loading">Fetching live state…</div></div>
<button id="send-btn" onclick="send()">Send</button> <div class="narr" id="ch1-narr"></div>
</div>
<div class="chapter">
<div class="num">Chapter 2</div>
<h2>The demand signal is real, not made up</h2>
<div class="lede">Chicago's Department of Buildings publishes every permit they issue. Below are the largest categories of construction filed in the last 30 days. If a staffer doesn't believe our numbers, they can verify at <a href="https://data.cityofchicago.org/Buildings/Building-Permits/ydr8-5enu" target="_blank" rel="noopener">data.cityofchicago.org</a>.</div>
<div id="ch2-permits"><div class="loading">Loading permit feed…</div></div>
</div>
<div class="chapter">
<div class="num">Chapter 3</div>
<h2>Where your own data would live</h2>
<div class="lede">The system stores data in labeled catalogs. Purple pills = synthetic stand-ins you'd swap for your real ATS/CRM/call-log exports. Blue pills = data the system generates about itself (playbooks, audit trails). Nothing else in the pipeline changes — only the source.</div>
<div id="ch3-datasets"><div class="loading">Enumerating catalog…</div></div>
<div class="narr">
<strong>The swap path.</strong> workers_500k → your ATS export (same schema shape). candidates → your CRM. call_log → your phone system's CDR. timesheets → your payroll export. Once ingested, every behavior you see on the dashboard applies to your real data. No re-training. No replatform.
</div> </div>
</div> </div>
<div class="ft">Powered by Lakehouse — Hybrid SQL + Vector Search across 500,000 embedded worker profiles</div>
<div class="chapter">
<div class="num">Chapter 4</div>
<h2>Watch the system rank candidates in real time</h2>
<div class="lede">This takes the most recent Chicago permit, derives the staffing need, pulls ranked candidates from the bench, and shows you why each one ranked. Everything below loaded in about 3 seconds against the live system.</div>
<div id="ch4-demo"><div class="loading">Running demo query…</div></div>
</div> </div>
<div class="chapter">
<div class="num">Chapter 5</div>
<h2>Every action compounds — the CRM-killer</h2>
<div class="lede">A CRM stores. This system compounds. Every successful fill, every no-show, every phone call becomes a re-ranking signal on the next query. Below is the live playbook memory state. The number grows as the app gets used.</div>
<div id="ch5-memory"><div class="loading">Reading playbook memory…</div></div>
</div>
<div class="chapter">
<div class="num">Chapter 6</div>
<h2>Try it yourself</h2>
<div class="lede">Type any staffing question. The system picks the right search path (smart-parse, semantic discovery, analytics), shows what it understood, and returns ranked results with memory signal.</div>
<div class="try-box">
<input type="text" id="try-q" placeholder="e.g. reliable forklift operators in Chicago with OSHA certs" onkeydown="if(event.key==='Enter')runTry()">
<button id="try-btn" onclick="runTry()">Ask</button>
<div id="try-out" style="margin-top:16px"></div>
</div>
</div>
</div>
<div class="footer">Lakehouse · demo instance · <a href="proof">architecture &amp; benchmarks</a></div>
<script> <script>
var P=location.pathname.replace(/\/console\/?$/,''); var P=location.pathname.indexOf('/lakehouse')>=0?'/lakehouse':'';
var A=location.origin+P; var A=location.origin+P;
var AC=['#1a2744','#1a3a2a','#2a1a3a','#3a2a1a','#1a3a3a','#2a2a1a','#3a1a2a','#1a2a3a'];
var briefData=null;
function api(path,body){ // DOM helpers — all dynamic content goes through these. No innerHTML
return fetch(A+path,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(body||{})}).then(function(r){return r.json()}) // anywhere in the script; every API-derived string passes through
// textContent so no injection path regardless of upstream data.
function el(tag, cls, text){
var e=document.createElement(tag);
if(cls) e.className=cls;
if(text!==undefined && text!==null) e.textContent=String(text);
return e;
}
function setAttrs(e, obj){
for(var k in obj) if(Object.prototype.hasOwnProperty.call(obj,k)) e.setAttribute(k,obj[k]);
return e;
} }
function el(tag,css,text){var e=document.createElement(tag);if(css)e.style.cssText=css;if(text)e.textContent=text;return e} function api(path, body){
function div(cls,text){var d=document.createElement('div');if(cls)d.className=cls;if(text)d.textContent=text;return d} return fetch(A+path,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(body||{})}).then(function(r){return r.json()});
}
function apiGet(path){
return fetch(A+path).then(function(r){return r.json()});
}
window.addEventListener('load',loadBrief); window.addEventListener('load',function(){
loadChapter1();
loadChapter2();
loadChapter3();
loadChapter4();
loadChapter5();
});
function loadBrief(){ // ─── Chapter 1 ────────────────────────────────────────────
api('/intelligence/brief').then(function(d){ function loadChapter1(){
briefData=d; Promise.all([
var container=document.getElementById('brief'); api('/intelligence/staffing_forecast',{}).catch(function(){return {}}),
container.textContent=''; apiGet('/api/vectors/playbook_memory/stats').catch(function(){return {}}),
document.getElementById('brief-time').textContent='Analyzed '+(d.pool.total||0).toLocaleString()+' profiles in '+(d.duration_ms||0)+'ms'; apiGet('/api/catalog/datasets').catch(function(){return []}),
var pool=d.pool; ]).then(function(all){
var relPct=pool.total?Math.round(pool.reliable/pool.total*100):0; var f=all[0]||{}, mem=all[1]||{}, ds=all[2]||[];
var elitePct=pool.total?Math.round(pool.elite/pool.total*100):0; var totalRows=0;
if(Array.isArray(ds)) ds.forEach(function(d){totalRows+=(d.row_count||0)});
var host=document.getElementById('ch1-stats');host.textContent='';
addStat(host,'$'+(f.total_cost||0).toLocaleString('en-US',{maximumFractionDigits:0}),'Construction pipeline',(f.permit_count||0)+' permits last 30 days','accent-b');
addStat(host,(f.total_estimated_workers||0).toLocaleString(),'Predicted worker demand','split across '+((f.forecast||[]).length)+' roles','accent-w');
addStat(host,(totalRows||0).toLocaleString(),'Rows under management',(Array.isArray(ds)?ds.length:0)+' datasets · real + synthetic','accent-a');
addStat(host,(mem.entries||0).toLocaleString(),'Playbooks remembered',(mem.total_names_endorsed||0)+' endorsed worker-tags','accent-l');
// Card 1: Workforce Pulse var narrEl=document.getElementById('ch1-narr');
var c1=makeCard('pulse','WORKFORCE PULSE','Your A-Team',pool.reliable.toLocaleString()+' workers with 80%+ reliability — '+relPct+'% of your pool'); narrEl.textContent='';
var bigRow=el('div','display:flex;gap:20px;margin-bottom:12px'); var critical=(f.critical_roles||0), tight=(f.tight_roles||0);
addBigNum(bigRow,pool.total.toLocaleString(),'Total Workers','#f0f6fc'); if(critical>0){
addBigNum(bigRow,relPct+'%','Reliable','#3fb950'); narrEl.appendChild(document.createTextNode('The system has already flagged '));
addBigNum(bigRow,elitePct+'%','Elite (90%+)','#58a6ff'); narrEl.appendChild(el('strong',null,critical+' role'+(critical!==1?'s':'')+' at critical coverage'));
c1.appendChild(bigRow); narrEl.appendChild(document.createTextNode(' — supply is below demand. A CRM would surface this only if someone ran the report. This system surfaces it before you log in.'));
if(d.archetypes&&d.archetypes.length){ } else if(tight>0){
var ar=div('arch-row'); narrEl.appendChild(document.createTextNode('The system flagged '));
var archColors={reliable:'#3fb950',communicator:'#58a6ff',flexible:'#d29922',leader:'#bc8cff',specialist:'#f0883e',improving:'#79c0ff',erratic:'#f85149',silent:'#484f58'}; narrEl.appendChild(el('strong',null,tight+' role'+(tight!==1?'s':'')+' as tight'));
d.archetypes.forEach(function(a){ narrEl.appendChild(document.createTextNode(' — watch for supply compression. Pre-emptive signal, no query required.'));
var pct=pool.total?Math.round(a.cnt/pool.total*100):0; } else {
var color=archColors[a.archetype]||'#484f58'; narrEl.textContent='No coverage gaps right now — bench exceeds predicted demand across all roles. The system is tracking this continuously; check back tomorrow and the numbers will have moved.';
var pill=el('span','background:'+color+'20;color:'+color+';border:1px solid '+color+'40;padding:3px 10px;border-radius:10px;font-size:11px;font-weight:500',a.archetype+' '+pct+'%');
pill.className='arch-pill';ar.appendChild(pill);
});
c1.appendChild(ar);
} }
container.appendChild(c1);
// Card 2: Geographic Bench
var c2=makeCard('bench','GEOGRAPHIC BENCH','Where You\'re Strong — and Thin','Reliable worker density by state');
var sorted=(d.bench||[]).slice().sort(function(a,b){return (b.total?b.reliable/b.total:0)-(a.total?a.reliable/a.total:0)});
sorted.forEach(function(s){
var pct=s.total?Math.round(s.reliable/s.total*100):0;
var color=pct>45?'#3fb950':pct>35?'#d29922':'#f85149';
var row=div('bench-row');
row.appendChild(el('span','width:24px;font-weight:600;color:#f0f6fc',s.state));
var bar=el('div','flex:1;height:14px;background:#161b22;border-radius:4px;overflow:hidden');
bar.appendChild(el('div','height:100%;border-radius:4px;background:'+color+';width:'+pct+'%;transition:width 0.8s'));
row.appendChild(bar);
row.appendChild(el('span','width:40px;text-align:right;font-size:11px;font-weight:600;color:'+color,pct+'%'));
row.appendChild(el('span','font-size:10px;color:#484f58;width:60px',s.total.toLocaleString()));
c2.appendChild(row);
});
var weakest=sorted[sorted.length-1];
if(weakest){
var wpct=weakest.total?Math.round(weakest.reliable/weakest.total*100):0;
c2.appendChild(makeCallout('Warning: '+weakest.state+' is your thinnest bench at '+wpct+'% reliable. If a client there ramps up, you may struggle to fill.','#1a1500','#854d0e'));
}
container.appendChild(c2);
// Card 3: Comeback Watch
if(d.gems&&d.gems.length){
var c3=makeCard('gems','COMEBACK WATCH',pool.improving.toLocaleString()+' Workers Improving','These workers crossed 80% reliability — they earned a second look');
d.gems.forEach(function(w){
addWorkerMini(c3,w.name,w.role+' · '+w.city+', '+w.state,'Reliability: '+Math.round(w.rel*100)+'% rising','#79c0ff','improving');
});
c3.appendChild(makeCallout('These workers were previously below threshold. The system detected their improvement automatically — no staffer had to track it.','#0d261a','#238636'));
container.appendChild(c3);
}
// Card 4: Risk Watch
if(d.risks&&d.risks.length){
var c4=makeCard('risk','RISK WATCH',pool.erratic.toLocaleString()+' Erratic + '+pool.silent_cnt.toLocaleString()+' Silent Workers','Low reliability + behavioral flags — review placement priority');
d.risks.forEach(function(w){
addWorkerMini(c4,w.name,w.role+' · '+w.city+', '+w.state,'Reliability: '+Math.round(w.rel*100)+'% · Responsiveness: '+Math.round(w.resp*100)+'%','#f85149',w.archetype);
});
c4.appendChild(makeCallout('Not firing — deprioritizing. Every failed placement costs time and client trust. The system flags these automatically so you don\'t learn the hard way.','#2d0d0d','#7f1d1d'));
container.appendChild(c4);
}
// Card 5: Ready & Waiting
if(d.untapped&&d.untapped.length){
var c5=makeCard('talent','READY & WAITING','Available + Reliable — Call These First','85%+ reliability and actively available right now');
d.untapped.forEach(function(w){
addWorkerMini(c5,w.name,w.role+' · '+w.city+', '+w.state,'Available: '+Math.round(w.avail*100)+'% · Reliable: '+Math.round(w.rel*100)+'%','#bc8cff','ready');
});
container.appendChild(c5);
}
// Card 6: Role Supply
if(d.supply&&d.supply.length){
var c6=makeCard('supply','ROLE SUPPLY','Workers by Role','Roles with high supply but low availability may indicate scheduling problems');
d.supply.slice(0,10).forEach(function(r){
var row=div('supply-row');
row.appendChild(el('span','color:#f0f6fc;font-weight:500',r.role));
var nums=el('span','color:#8b949e;font-size:11px');
nums.appendChild(el('strong','color:#f0f6fc',r.supply.toLocaleString()));
nums.appendChild(document.createTextNode(' total · '));
nums.appendChild(el('strong','color:#f0f6fc',r.available.toLocaleString()));
nums.appendChild(document.createTextNode(' available · rel '+Math.round(r.avg_rel*100)+'%'));
row.appendChild(nums);c6.appendChild(row);
});
container.appendChild(c6);
}
setupPrompts(d);
}).catch(function(e){ }).catch(function(e){
document.getElementById('brief').textContent='Failed to load brief: '+e.message; var h=document.getElementById('ch1-stats');h.textContent='';h.appendChild(el('div','err','Live state unavailable: '+(e.message||e)));
});
document.getElementById('hdr-time').textContent='Snapshot · '+new Date().toLocaleTimeString();
}
function addStat(host,n,l,sub,cls){
var d=el('div','card stat-lg '+(cls||''));
d.appendChild(el('div','n',n));
d.appendChild(el('div','l',l));
d.appendChild(el('div','sub',sub||''));
host.appendChild(d);
}
// ─── Chapter 2 ────────────────────────────────────────────
function loadChapter2(){
api('/intelligence/market',{}).then(function(r){
var host=document.getElementById('ch2-permits');host.textContent='';
if(r.error){host.appendChild(el('div','err','Permit feed error: '+r.error));return}
var byType=r.by_type||[];
if(byType.length===0){host.appendChild(el('div','narr','No permits returned by the Chicago API right now.'));return}
byType.slice(0,8).forEach(function(t){
var det=document.createElement('details');
var summary=document.createElement('summary');
summary.appendChild(el('span','pill real','LIVE · DATA.CITYOFCHICAGO.ORG'));
summary.appendChild(document.createTextNode(' '));
var wt=el('strong',null,t.work_type||'General construction');
summary.appendChild(wt);
summary.appendChild(document.createTextNode(' · '+(t.permits||0)+' permits · $'+(t.total_cost||0).toLocaleString('en-US',{maximumFractionDigits:0})+' · est '+(t.estimated_workers||0)+' workers needed'));
det.appendChild(summary);
var body=el('div','body');
body.appendChild(document.createTextNode('Likely roles: '+((t.needed_roles||[]).join(', ')||'—')+'. '));
body.appendChild(document.createTextNode('At industry heuristic (~$150K per worker), this work type alone implies '));
body.appendChild(el('strong',null,(t.estimated_workers||0)+' staffing opportunities'));
body.appendChild(document.createTextNode(' over the next 30-60 days.'));
det.appendChild(body);
host.appendChild(det);
});
}).catch(function(e){
var h=document.getElementById('ch2-permits');h.textContent='';h.appendChild(el('div','err',e.message||String(e)));
}); });
} }
function makeCard(cls,label,headline,sub){ // ─── Chapter 3 ────────────────────────────────────────────
var d=div('bcard '+cls); function loadChapter3(){
d.appendChild(div('label',label));d.appendChild(div('headline',headline));d.appendChild(div('sub',sub));return d; apiGet('/api/catalog/datasets').then(function(ds){
} var host=document.getElementById('ch3-datasets');host.textContent='';
function addBigNum(parent,num,label,color){ if(!Array.isArray(ds)){host.appendChild(el('div','err','No datasets returned.'));return}
var d=el('div',''); var bigOnes=ds.filter(function(d){return (d.row_count||0)>=500}).sort(function(a,b){return (b.row_count||0)-(a.row_count||0)});
var n=el('div','font-size:36px;font-weight:800;line-height:1;color:'+color,num); var playbookKeys=['successful_playbooks','playbook_memory','kb_'];
d.appendChild(n);d.appendChild(el('div','font-size:10px;color:#484f58;margin-top:2px',label));parent.appendChild(d); bigOnes.slice(0,12).forEach(function(d){
} var row=el('div','row');
function addWorkerMini(parent,name,detail,extra,color,badge){ var left=document.createElement('div');left.style.flex='1';left.style.minWidth='0';
var w=div('wk'); var t=el('div','title');
var av=el('div','width:32px;height:32px;border-radius:8px;display:flex;align-items:center;justify-content:center;font-weight:700;font-size:12px;color:#f0f6fc;flex-shrink:0;background:'+AC[Math.floor(Math.random()*AC.length)]); var pillCls='synth', pillLabel='SWAP FOR YOUR DATA';
av.textContent=(name||'').split(' ').map(function(n){return(n[0]||'').toUpperCase()}).slice(0,2).join(''); if(playbookKeys.some(function(k){return (d.name||'').indexOf(k)>=0})){pillCls='flow';pillLabel='SYSTEM-GENERATED'}
var info=div('info'); else if((d.name||'').indexOf('threat_')===0){pillCls='flow';pillLabel='SYSTEM-GENERATED'}
info.appendChild(div('nm',name));info.appendChild(div('det',detail)); t.appendChild(el('span','pill '+pillCls,pillLabel));
if(extra){var ex=div('det');ex.style.color='#58a6ff';ex.textContent=extra;info.appendChild(ex)} t.appendChild(document.createTextNode(' '+(d.name||'(unnamed)')));
w.appendChild(av);w.appendChild(info); var m=el('div','meta',(d.columns?d.columns.length+' columns · ':'')+(d.objects?d.objects.length+' parquet file'+(d.objects.length!==1?'s':''):''));
if(badge){var b=el('span','padding:2px 8px;border-radius:8px;font-size:10px;font-weight:600;background:'+(color||'#484f58')+'20;color:'+(color||'#484f58'),badge);w.appendChild(b)} left.appendChild(t);left.appendChild(m);
parent.appendChild(w); var v=el('div','val',(d.row_count||0).toLocaleString()+' rows');
} row.appendChild(left);row.appendChild(v);host.appendChild(row);
function makeCallout(text,bg,border){ });
return el('div','font-size:11px;color:#8b949e;margin-top:8px;padding:8px 10px;background:'+(bg||'#0d1117')+';border:1px solid '+(border||'#1b2130')+';border-radius:6px;font-style:italic',text); }).catch(function(e){
var h=document.getElementById('ch3-datasets');h.textContent='';h.appendChild(el('div','err','Catalog unavailable: '+(e.message||e)));
});
} }
function setupPrompts(data){ // ─── Chapter 4 ────────────────────────────────────────────
var container=document.getElementById('prompts');container.textContent=''; function loadChapter4(){
var prompts=[]; api('/intelligence/permit_contracts',{}).then(function(r){
if(data.gems&&data.gems[0])prompts.push('Find someone like '+data.gems[0].name+' but in OH'); var host=document.getElementById('ch4-demo');host.textContent='';
prompts.push('Who could handle industrial electrical work?'); if(!r.contracts||r.contracts.length===0){host.appendChild(el('div','loading','No contracts returned.'));return}
prompts.push('What if we lose our top 5 forklift operators?'); var c=r.contracts[0];
prompts.push('Which workers should we stop placing?'); var p=c.permit||{}, prop=c.proposed||{}, tl=c.timeline||{};
if(data.supply&&data.supply.length>2){ var card=el('div','card accent-b');
var under=data.supply.slice().sort(function(a,b){return (a.available/a.supply)-(b.available/b.supply)})[0]; var title=el('div',null,'$'+(p.cost||0).toLocaleString()+' · '+(p.work_type||''));
if(under)prompts.push('Show me available '+under.role+'s'); title.style.cssText='font-size:15px;font-weight:600;color:#e6edf3';
card.appendChild(title);
var addr=el('div',null,(p.address||'')+' · Chicago, IL · filed '+(p.issue_date||''));
addr.style.cssText='color:#8b949e;font-size:12px;margin-top:2px';
card.appendChild(addr);
card.appendChild(el('div','step-label','STEP 1 · Derive staffing need'));
var s1=el('div','step-body');
s1.appendChild(document.createTextNode('Industry heuristic: ~1 worker per $150K of permit cost, capped 2-8. Resulting contract: '));
s1.appendChild(el('strong',null,(prop.count||0)+'× '+(prop.role||'')));
s1.appendChild(document.createTextNode(' in Chicago, IL.'));
card.appendChild(s1);
card.appendChild(el('div','step-label','STEP 2 · Narrow via SQL ('+((prop.pool_size||0).toLocaleString())+' candidates match)'));
card.appendChild(el('div','step-body',"role = '"+(prop.role||'')+"' AND state = 'IL' AND city = 'Chicago' AND CAST(availability AS DOUBLE) > 0.5"));
card.appendChild(el('div','step-label','STEP 3 · Rank via semantic match + playbook boost'));
host.appendChild(card);
var list=document.createElement('div');list.style.marginTop='6px';
(prop.candidates||[]).slice(0,5).forEach(function(cand,i){
var w=el('div','worker');
var initials=(cand.name||'?').split(' ').map(function(s){return (s[0]||'').toUpperCase()}).join('').substring(0,2);
w.appendChild(el('div','av',initials));
var info=el('div','info');
var nm=el('div','nm',cand.name||cand.doc_id||'?');
if((cand.playbook_boost||0)>0){
var ncit=(cand.playbook_citations||[]).length;
nm.appendChild(el('span','boost-chip','Endorsed · '+ncit+' past fill'+(ncit!==1?'s':'')));
} }
prompts.push('Find me a warehouse worker available today near Nashville'); info.appendChild(nm);
prompts.push('Find bilingual workers with leadership skills'); var why=cand.doc_id+' · '+(cand.playbook_boost>0?'boosted +'+cand.playbook_boost.toFixed(3)+' by memory · ':'')+'semantic score '+(cand.score||0).toFixed(3);
prompts.forEach(function(p){ info.appendChild(el('div','why',why));
var btn=el('button','',p);btn.className='prompt-btn'; w.appendChild(info);
btn.onclick=function(){document.getElementById('q').value=p;send()}; w.appendChild(el('div','score','#'+(i+1)));
container.appendChild(btn); list.appendChild(w);
});
card.appendChild(list);
if(c.discovered_pattern||c.pattern_matched>0){
var mem=el('div','mem-chip');
mem.appendChild(el('span','l','MEMORY ('+(c.pattern_matched||0)+' past playbooks):'));
mem.appendChild(document.createTextNode(' '+(c.discovered_pattern||'memory is sparse for this role+geo')));
card.appendChild(mem);
}
if(tl.days_to_deadline!==undefined){
var tm=document.createElement('div');
tm.style.cssText='margin-top:12px;padding-top:10px;border-top:1px solid #171d27;font-size:12px;color:#8b949e';
tm.appendChild(el('strong',null,'Timeline. '));
tm.childNodes[0].style && (tm.childNodes[0].style.color='#c9d1d9');
var dd=tl.days_to_deadline;
var when=(dd<=0?Math.abs(dd)+' days overdue':dd+' days out');
tm.appendChild(document.createTextNode('Staffing window opens '+(tl.staffing_window_opens||'')+' ('+when+'). Construction starts '+(tl.estimated_construction_start||'')+'. Urgency: '));
tm.appendChild(el('strong',null,(tl.urgency||'').toUpperCase()));
tm.appendChild(document.createTextNode('.'));
card.appendChild(tm);
}
var note=el('div','narr');
note.appendChild(el('strong',null,'This is the CRM gap. '));
note.appendChild(document.createTextNode('A CRM would tell you the permit exists. It would NOT narrow to matching candidates, would NOT boost past-successful workers, would NOT flag patterns across similar past fills, would NOT count down to the deadline. All that happens here, pre-computed, before the staffer clicks anything.'));
host.appendChild(note);
}).catch(function(e){
var h=document.getElementById('ch4-demo');h.textContent='';h.appendChild(el('div','err','Demo failed: '+(e.message||e)));
}); });
} }
// ─── Chat ─── // ─── Chapter 5 ────────────────────────────────────────────
function send(){ function loadChapter5(){
var input=document.getElementById('q'); apiGet('/api/vectors/playbook_memory/stats').then(function(mem){
var q=input.value.trim();if(!q)return; var host=document.getElementById('ch5-memory');host.textContent='';
input.value=''; var card=el('div','card accent-l');
var chat=document.getElementById('chat'); var big=el('div',null,(mem.entries||0).toLocaleString());
var userMsg=div('msg user');var bubble=div('bubble',q);userMsg.appendChild(bubble);chat.appendChild(userMsg); big.style.cssText='font-size:30px;font-weight:800;color:#e6edf3;line-height:1';
var thinking=el('div','display:flex;align-items:center;gap:8px;color:#58a6ff;font-size:12px;padding:12px'); card.appendChild(big);
thinking.appendChild(document.createTextNode('Querying 500K profiles')); var lb=el('div',null,'Playbooks in memory');
var dots=el('span','');dots.className='dots'; lb.style.cssText='font-size:10px;color:#545d68;text-transform:uppercase;letter-spacing:1.2px;margin-top:8px;font-weight:600';
dots.appendChild(el('span','','.'));dots.appendChild(el('span','','.'));dots.appendChild(el('span','','.')); card.appendChild(lb);
thinking.appendChild(dots);chat.appendChild(thinking);chat.scrollTop=chat.scrollHeight; var sub=el('div',null,(mem.total_names_endorsed||0).toLocaleString()+' endorsed worker-tags · each one boosts future matching queries');
document.getElementById('send-btn').disabled=true; sub.style.cssText='font-size:12px;color:#8b949e;margin-top:4px';
card.appendChild(sub);
host.appendChild(card);
if(mem.sample&&mem.sample.length){
var hdr=document.createElement('div');
hdr.style.cssText='color:#545d68;font-size:11px;text-transform:uppercase;letter-spacing:1.4px;font-weight:600;margin:14px 0 8px';
hdr.textContent='Sample playbooks held in memory right now';
host.appendChild(hdr);
mem.sample.slice(0,5).forEach(function(pb){
var row=el('div','row');
var left=document.createElement('div');left.style.flex='1';left.style.minWidth='0';
left.appendChild(el('div','title',pb.operation||'(no operation)'));
left.appendChild(el('div','meta',(pb.city||'')+', '+(pb.state||'')+' · endorsed: '+((pb.endorsed||[]).slice(0,3).join(', ')||'—')));
row.appendChild(left);host.appendChild(row);
});
}
var narr=el('div','narr');
narr.appendChild(el('strong',null,'What this means for a staffer. '));
narr.appendChild(document.createTextNode('When you log a successful fill, this counter grows by one. Next time someone searches for a similar role+city, the system finds your past pick and boosts them to the top. Over months, the system becomes a record of who has actually worked out — not just who is on the roster.'));
host.appendChild(narr);
}).catch(function(e){
var h=document.getElementById('ch5-memory');h.textContent='';h.appendChild(el('div','err','Memory unavailable: '+(e.message||e)));
});
}
// ─── Chapter 6 ────────────────────────────────────────────
function runTry(){
var q=document.getElementById('try-q').value.trim();if(!q)return;
var btn=document.getElementById('try-btn'),out=document.getElementById('try-out');
btn.disabled=true;btn.textContent='Thinking…';out.textContent='';out.appendChild(el('div','loading','Running…'));
api('/intelligence/chat',{message:q}).then(function(d){ api('/intelligence/chat',{message:q}).then(function(d){
thinking.remove();document.getElementById('send-btn').disabled=false; btn.disabled=false;btn.textContent='Ask';out.textContent='';
var msg=div('msg system');var bbl=div('bubble'); var card=el('div','card accent-b');
// Steps
if(d.queries_run&&d.queries_run.length){
var steps=div('steps');
d.queries_run.forEach(function(s){steps.appendChild(el('div','font-size:11px;color:#58a6ff;padding:2px 0','→ '+s))});
bbl.appendChild(steps);
}
bbl.appendChild(div('summary',d.summary||''));
// Type-specific rendering
if(d.type==='similar'&&d.source){
var sc=div('source-card');
sc.appendChild(el('div','font-size:10px;color:#d29922;text-transform:uppercase;letter-spacing:1px;margin-bottom:4px','Source Worker'));
sc.appendChild(el('div','font-weight:700;color:#f0f6fc;font-size:14px',d.source.name));
sc.appendChild(el('div','font-size:12px;color:#d29922',d.source.role+' · '+d.source.city+', '+d.source.state+' · Reliability: '+Math.round(d.source.rel*100)+'%'));
sc.appendChild(el('div','font-size:11px;color:#8b949e;margin-top:4px','Skills: '+(d.source.skills||'none')+' · Archetype: '+(d.source.archetype||'unknown')));
bbl.appendChild(sc);
renderResults(bbl,d.results);
if(d.sql_matches)bbl.appendChild(el('div','font-size:10px;color:#484f58;margin-top:6px','Filtered pool: '+d.sql_matches.toLocaleString()+' workers'));
}
else if(d.type==='discovery'){
renderResults(bbl,d.results);
if(d.note)bbl.appendChild(el('div','font-size:11px;color:#8b949e;font-style:italic;margin-top:8px;padding:8px;background:#0d1117;border-radius:6px',d.note));
}
else if(d.type==='whatif'){
var sevColors={HIGH:['#7f1d1d','#fca5a5'],MEDIUM:['#854d0e','#fcd34d'],LOW:['#238636','#86efac']};
var sc2=sevColors[d.risk_level]||sevColors.MEDIUM;
var sev=el('span','display:inline-block;padding:3px 10px;border-radius:8px;font-size:11px;font-weight:700;margin-left:8px;background:'+sc2[0]+';color:'+sc2[1],'RISK: '+d.risk_level);
bbl.querySelector('.summary').appendChild(sev);
bbl.appendChild(el('div','font-size:11px;color:#f85149;margin:8px 0 4px;font-weight:600','Workers you\'d lose:'));
(d.lost||[]).forEach(function(w){
var rc=div('result-card');rc.style.borderLeft='3px solid #f85149';
rc.appendChild(el('div','font-weight:600;color:#f0f6fc;font-size:13px',w.name));
rc.appendChild(el('div','font-size:11px;color:#8b949e;margin-top:2px',w.role+' · '+w.city+', '+w.state+' · Reliability: '+Math.round(w.rel*100)+'%'));
bbl.appendChild(rc);
});
bbl.appendChild(el('div','font-size:11px;color:#3fb950;margin:12px 0 4px;font-weight:600','Remaining bench: '+d.reliable_remaining+' reliable workers across '+(d.bench||[]).length+' states ('+d.total_in_role+' total in role)'));
(d.bench||[]).forEach(function(b){
var row=el('div','display:flex;justify-content:space-between;padding:4px 8px;font-size:12px');
row.appendChild(el('span','color:#f0f6fc',b.state+' — '+b.total+' total'));
row.appendChild(el('span','color:'+(b.reliable>5?'#3fb950':'#f85149'),b.reliable+' reliable'));
bbl.appendChild(row);
});
}
else if(d.type==='risk'){
(d.results||[]).forEach(function(w){
var rc=div('result-card');rc.style.borderLeft='3px solid #f85149';
rc.appendChild(el('div','font-weight:600;color:#f0f6fc;font-size:13px',w.name));
rc.appendChild(el('div','font-size:11px;color:#8b949e;margin-top:2px',w.role+' · '+w.city+', '+w.state));
rc.appendChild(el('div','font-size:10px;color:#f85149;margin-top:4px','Reliability: '+Math.round(w.rel*100)+'% · Responsiveness: '+Math.round(w.resp*100)+'% · Compliance: '+Math.round(w.compl*100)+'% · '+w.archetype));
bbl.appendChild(rc);
});
if(d.total_flagged)bbl.appendChild(el('div','font-size:10px;color:#484f58;margin-top:6px',d.total_flagged.toLocaleString()+' total workers match this risk profile'));
}
else if(d.type==='smart_search'){
// Show what the system understood
if(d.understood&&d.understood.length){ if(d.understood&&d.understood.length){
var tags=el('div','display:flex;gap:6px;flex-wrap:wrap;margin-bottom:10px'); var tags=document.createElement('div');tags.style.cssText='display:flex;gap:6px;flex-wrap:wrap;margin-bottom:10px';
d.understood.forEach(function(u){ d.understood.forEach(function(u){
tags.appendChild(el('span','padding:3px 10px;border-radius:10px;font-size:11px;background:#1a274420;color:#58a6ff;border:1px solid #1a274480',u)); var tg=el('span',null,u);
tg.style.cssText='padding:3px 10px;border-radius:10px;font-size:11px;background:#1a274420;color:#58a6ff;border:1px solid #1a274480';
tags.appendChild(tg);
}); });
bbl.appendChild(tags); card.appendChild(tags);
} }
// Show SQL results with zip codes, availability, skills var sum=el('div',null,d.summary||'(no summary)');
if(d.sql_results&&d.sql_results.length){ sum.style.cssText='color:#c9d1d9;font-size:13px;margin-bottom:10px';
d.sql_results.forEach(function(w){ card.appendChild(sum);
var rc=div('result-card');
rc.appendChild(el('div','font-weight:600;color:#f0f6fc;font-size:14px',w.name)); if(d.pattern_playbooks_matched>0&&d.discovered_pattern){
var locParts=[w.role,w.city+', '+w.state]; var mem=el('div','mem-chip');
if(w.zip)locParts.push('ZIP: '+w.zip); mem.appendChild(el('span','l','MEMORY ('+d.pattern_playbooks_matched+' playbooks):'));
rc.appendChild(el('div','font-size:12px;color:#8b949e;margin-top:2px',locParts.join(' · '))); mem.appendChild(document.createTextNode(' '+d.discovered_pattern));
// Metrics row card.appendChild(mem);
var metrics=[];
if(w.avail!==undefined)metrics.push('Available: '+Math.round(w.avail*100)+'%');
if(w.rel!==undefined)metrics.push('Reliable: '+Math.round(w.rel*100)+'%');
if(w.archetype)metrics.push(w.archetype);
if(metrics.length)rc.appendChild(el('div','font-size:11px;color:#58a6ff;margin-top:4px',metrics.join(' · ')));
// Skills + certs
if(w.skills||w.certifications){
var tagRow=el('div','display:flex;gap:4px;flex-wrap:wrap;margin-top:6px');
if(w.skills)(w.skills||'').split(',').forEach(function(s){s=s.trim();if(s)tagRow.appendChild(el('span','padding:2px 8px;background:#1a2744;color:#58a6ff;border-radius:8px;font-size:10px',s))});
if(w.certifications)(w.certifications||'').split(',').forEach(function(c){c=c.trim();if(c&&c!=='none')tagRow.appendChild(el('span','padding:2px 8px;background:#1a3a2a;color:#3fb950;border-radius:8px;font-size:10px',c))});
rc.appendChild(tagRow);
}
bbl.appendChild(rc);
});
}
// Also show vector results if different
if(d.vector_results&&d.vector_results.length&&(!d.sql_results||!d.sql_results.length)){
renderResults(bbl,d.vector_results);
}
}
else if(d.type==='search'||d.type==='answer'){
renderResults(bbl,d.results||d.sources||[]);
}
else if(d.type==='error'){
bbl.appendChild(el('div','color:#f85149;font-size:13px',d.summary));
} }
if(d.duration_ms){ var workers=d.sql_results||d.vector_results||d.results||[];
var meta=div('meta'); workers.slice(0,5).forEach(function(w,i){
meta.appendChild(el('span','margin-right:12px',d.duration_ms+'ms')); var row=el('div','worker');
if(d.sql_matches)meta.appendChild(el('span','',d.sql_matches.toLocaleString()+' records scanned')); var nm=w.name||(w.text||'').split('—')[0].trim()||w.doc_id||'?';
bbl.appendChild(meta); var initials=nm.split(' ').map(function(s){return (s[0]||'').toUpperCase()}).join('').substring(0,2);
row.appendChild(el('div','av',initials));
var info=el('div','info');
var n=el('div','nm',nm);
if((w.playbook_boost||0)>0){
n.appendChild(el('span','boost-chip','Endorsed · '+((w.playbook_citations||[]).length||'?')+' past fill(s)'));
} }
msg.appendChild(bbl);chat.appendChild(msg);chat.scrollTop=chat.scrollHeight; info.appendChild(n);
var bits=[];
if(w.role) bits.push(w.role);
if(w.city&&w.state) bits.push(w.city+', '+w.state);
if(w.rel!==undefined) bits.push('reliability '+Math.round(w.rel*100)+'%');
if(w.avail!==undefined) bits.push('availability '+Math.round(w.avail*100)+'%');
info.appendChild(el('div','why',bits.join(' · ')||'AI semantic match'));
row.appendChild(info);
row.appendChild(el('div','score','#'+(i+1)));
card.appendChild(row);
});
var meta=el('div',null,(d.duration_ms||0)+'ms · type='+(d.type||'smart_search'));
meta.style.cssText='color:#545d68;font-size:11px;margin-top:10px';
card.appendChild(meta);
out.appendChild(card);
}).catch(function(e){ }).catch(function(e){
thinking.remove();document.getElementById('send-btn').disabled=false; btn.disabled=false;btn.textContent='Ask';
var msg=div('msg system');var bbl=div('bubble');bbl.style.borderColor='#f85149'; out.textContent='';out.appendChild(el('div','err',e.message||String(e)));
bbl.appendChild(el('div','color:#f85149','Error: '+e.message));
msg.appendChild(bbl);chat.appendChild(msg);
}); });
} }
</script>
function renderResults(parent,results){ </body></html>
if(!results||!results.length)return;
results.forEach(function(r){
var text=r.text||r.chunk_text||'';
var parts=text.split(/\u2014|\u2013|\u002D{2,}/);
var name=parts[0]?parts[0].trim():(r.name||r.doc_id||'Worker');
var rest=parts[1]?parts[1].trim():'';
var roleM=rest.match(/^(.+?)\s+in\s+(.+?)\./);
var skillM=rest.match(/Skills:\s*([^.]+)/);
var certM=rest.match(/Cert(?:s|ifications)?:\s*([^.]+)/);
var relM=rest.match(/Reliability:\s*([\d.]+)/);
var archM=rest.match(/Archetype:\s*(\w+)/);
var rc=div('result-card');
rc.appendChild(el('div','font-weight:600;color:#f0f6fc;font-size:13px',name));
if(roleM)rc.appendChild(el('div','font-size:11px;color:#8b949e;margin-top:2px',roleM[1]+' · '+roleM[2]));
var scoreParts=[];
if(r.score)scoreParts.push('Match: '+Math.round(r.score*100)+'%');
if(relM)scoreParts.push('Reliability: '+Math.round(relM[1]*100)+'%');
if(archM)scoreParts.push(archM[1]);
if(scoreParts.length)rc.appendChild(el('div','font-size:10px;color:#58a6ff;margin-top:4px',scoreParts.join(' · ')));
if(skillM||certM){
var tags=el('div','display:flex;gap:4px;flex-wrap:wrap;margin-top:6px');
if(skillM)skillM[1].split(/[|,]/).forEach(function(s){if(s.trim()){tags.appendChild(el('span','padding:2px 8px;background:#1a2744;color:#58a6ff;border-radius:8px;font-size:10px',s.trim()))}});
if(certM)certM[1].split(/[|,]/).forEach(function(c){if(c.trim()&&c.trim()!=='none'){tags.appendChild(el('span','padding:2px 8px;background:#1a3a2a;color:#3fb950;border-radius:8px;font-size:10px',c.trim()))}});
rc.appendChild(tags);
}
parent.appendChild(rc);
});
}
</script></body></html>
mcp-server/context7_bridge.ts
+178
View File
@@ -0,0 +1,178 @@
// Phase 45 slice 2 — context7 HTTP bridge.
//
// Exposes an HTTP surface the Rust gateway consumes to check external
// doc drift on playbooks. Wraps context7's public API:
//
// https://context7.com/api/v1/search?query=<name> → resolve
// https://context7.com/api/v1/<lib-id>/cli?type=txt&tokens=N → docs
//
// Own port so a failure here never tips over the mcp-server on :3700.
// Cache is in-memory (5 min TTL) — context7 rate-limits by IP, and
// gateway drift-checks are the hot caller.
//
// Endpoints:
// GET /health health + cache stats
// GET /docs/:tool resolve + fetch + return descriptor
// GET /docs/:tool/diff?since=X return drift vs content hash X
// GET /cache dump current cache (debugging)
import { createHash } from "node:crypto";
const PORT = Number(process.env.CONTEXT7_BRIDGE_PORT ?? 3900);
const CONTEXT7_BASE = (process.env.CONTEXT7_BASE_URL ?? "https://context7.com/api/v1").replace(/\/+$/, "");
const TOKENS_PER_FETCH = 1500; // enough for a drift-meaningful slice, not so much we hammer
const CACHE_TTL_MS = 5 * 60 * 1000;
interface CachedDoc {
tool: string;
library_id: string; // context7 canonical, e.g. "/docker/docs"
title: string;
last_updated: string | null;
snippet_hash: string; // SHA-256 of first TOKENS_PER_FETCH of fetched docs
docs_preview: string; // first 400 chars, returned in responses for display
retrieved_at: string; // ISO when WE last fetched
source_url: string; // human-facing context7 URL
}
const cache = new Map<string, { entry: CachedDoc; at: number }>();
function normalizeTool(s: string): string {
return s.trim().toLowerCase();
}
async function resolveLibraryId(tool: string): Promise<{ id: string; title: string; last_updated: string | null } | null> {
const r = await fetch(`${CONTEXT7_BASE}/search?query=${encodeURIComponent(tool)}`, {
signal: AbortSignal.timeout(10000),
});
if (!r.ok) throw new Error(`context7 search ${r.status}: ${await r.text().catch(() => "?")}`);
const j = await r.json() as any;
const results = Array.isArray(j.results) ? j.results : [];
// First result is top-ranked; context7 sorts by its internal score.
// Skip "state != finalized" to avoid unstable previews.
const pick = results.find((x: any) => x?.state === "finalized") ?? results[0];
if (!pick?.id) return null;
return {
id: String(pick.id),
title: String(pick.title ?? tool),
last_updated: typeof pick.lastUpdateDate === "string" ? pick.lastUpdateDate : null,
};
}
async function fetchDocsText(libraryId: string): Promise<string> {
// Strip the leading slash on library_id so the URL doesn't have a
// double slash. context7 returns ids like "/docker/docs" but the
// fetch path wants "docker/docs".
const cleanId = libraryId.replace(/^\/+/, "");
const url = `${CONTEXT7_BASE}/${cleanId}?type=txt&tokens=${TOKENS_PER_FETCH}`;
const r = await fetch(url, { signal: AbortSignal.timeout(20000) });
if (!r.ok) throw new Error(`context7 fetch ${r.status} on ${cleanId}: ${await r.text().catch(() => "?")}`);
return await r.text();
}
function hashContent(s: string): string {
return createHash("sha256").update(s).digest("hex").slice(0, 16);
}
async function getCurrent(tool: string): Promise<CachedDoc> {
const key = normalizeTool(tool);
const cached = cache.get(key);
if (cached && Date.now() - cached.at < CACHE_TTL_MS) {
return cached.entry;
}
const resolved = await resolveLibraryId(key);
if (!resolved) {
throw new Error(`no context7 library found for '${tool}'`);
}
const text = await fetchDocsText(resolved.id);
const entry: CachedDoc = {
tool: key,
library_id: resolved.id,
title: resolved.title,
last_updated: resolved.last_updated,
snippet_hash: hashContent(text),
docs_preview: text.slice(0, 400),
retrieved_at: new Date().toISOString(),
source_url: `https://context7.com${resolved.id.startsWith("/") ? resolved.id : "/" + resolved.id}`,
};
cache.set(key, { entry, at: Date.now() });
return entry;
}
function jsonResponse(body: unknown, status: number = 200): Response {
return new Response(JSON.stringify(body), {
status,
headers: { "content-type": "application/json" },
});
}
Bun.serve({
port: PORT,
hostname: "0.0.0.0",
async fetch(req) {
const url = new URL(req.url);
if (url.pathname === "/health") {
return jsonResponse({
status: "ok",
cache_size: cache.size,
context7_base: CONTEXT7_BASE,
ttl_ms: CACHE_TTL_MS,
});
}
if (url.pathname === "/cache") {
const dump = Array.from(cache.entries()).map(([k, v]) => ({
tool: k,
library_id: v.entry.library_id,
snippet_hash: v.entry.snippet_hash,
retrieved_at: v.entry.retrieved_at,
age_ms: Date.now() - v.at,
}));
return jsonResponse({ entries: dump });
}
// GET /docs/:tool — return current descriptor (fetches + caches)
const m1 = url.pathname.match(/^\/docs\/([^/]+)$/);
if (m1 && req.method === "GET") {
const tool = decodeURIComponent(m1[1]);
try {
const entry = await getCurrent(tool);
return jsonResponse(entry);
} catch (e) {
return jsonResponse({ error: (e as Error).message }, 404);
}
}
// GET /docs/:tool/diff?since=hash — compare current vs recorded hash
const m2 = url.pathname.match(/^\/docs\/([^/]+)\/diff$/);
if (m2 && req.method === "GET") {
const tool = decodeURIComponent(m2[1]);
const since = url.searchParams.get("since");
if (!since) {
return jsonResponse({ error: "query param 'since' (snippet_hash) required" }, 400);
}
try {
const entry = await getCurrent(tool);
const drifted = entry.snippet_hash !== since;
return jsonResponse({
tool: entry.tool,
drifted,
previous_snippet_hash: since,
current_snippet_hash: entry.snippet_hash,
library_id: entry.library_id,
title: entry.title,
last_updated_upstream: entry.last_updated,
retrieved_at: entry.retrieved_at,
source_url: entry.source_url,
docs_preview: drifted ? entry.docs_preview : null,
});
} catch (e) {
return jsonResponse({ error: (e as Error).message }, 404);
}
}
return jsonResponse({ error: "not found", paths: ["/health", "/docs/:tool", "/docs/:tool/diff?since=HASH", "/cache"] }, 404);
},
});
console.log(`[context7-bridge] listening on :${PORT} (context7 base: ${CONTEXT7_BASE})`);
mcp-server/index.ts
+1137 -516
View File
File diff suppressed because it is too large Load Diff
mcp-server/observer.ts
+108 -8
View File
@@ -20,6 +20,9 @@
const GATEWAY = process.env.GATEWAY_URL || "http://localhost:3700"; const GATEWAY = process.env.GATEWAY_URL || "http://localhost:3700";
const LAKEHOUSE = process.env.LAKEHOUSE_URL || "http://localhost:3100"; const LAKEHOUSE = process.env.LAKEHOUSE_URL || "http://localhost:3100";
const CYCLE_SECS = parseInt(process.env.OBSERVER_CYCLE || "30"); const CYCLE_SECS = parseInt(process.env.OBSERVER_CYCLE || "30");
// Phase 24 — observer now listens on an HTTP port for external ops
// (scenarios bypass the MCP:3700 layer by design). Default 3800.
const OBSERVER_PORT = parseInt(process.env.OBSERVER_PORT || "3800");
// ─── Observed operation log ─── // ─── Observed operation log ───
@@ -31,10 +34,32 @@ interface ObservedOp {
duration_ms: number; duration_ms: number;
output_summary: string; output_summary: string;
error?: string; error?: string;
// Phase 24 — optional provenance so error analyzer and playbook
// builder can differentiate MCP-layer ops from scenario-sourced
// events. Scenarios set source="scenario" + staffer_id + sig_hash.
source?: "mcp" | "scenario";
staffer_id?: string;
sig_hash?: string;
event_kind?: string;
role?: string;
city?: string;
state?: string;
count?: number;
rescue_attempted?: boolean;
rescue_succeeded?: boolean;
} }
const recentOps: ObservedOp[] = []; const recentOps: ObservedOp[] = [];
// Phase 24 — external ingest path. Scenarios POST outcome summaries
// here so the observer's analyzer + playbook builder see them. Called
// from the Bun.serve() handler below. Same ring buffer as the MCP-
// wrapped path so downstream loops don't need to know the source.
export function recordExternalOp(op: ObservedOp): void {
recentOps.push({ ...op, source: op.source ?? "scenario" });
if (recentOps.length > 2000) recentOps.shift();
}
// ─── Wrapped gateway caller — every call gets observed ─── // ─── Wrapped gateway caller — every call gets observed ───
export async function observed( export async function observed(
@@ -93,16 +118,22 @@ function summarize(data: any): string {
return JSON.stringify(data).slice(0, 100); return JSON.stringify(data).slice(0, 100);
} }
// Phase 24 honesty fix — the old persistOp used /ingest/file which
// REPLACES the dataset (flagged in feedback_ingest_replace_semantics.md).
// Every op silently wiped all prior ops. Now we append a JSONL line to
// data/_observer/ops.jsonl so the historical trace is durable. The
// observer analyzer + playbook builder read from this file when it
// outgrows the 2000-entry in-memory ring.
async function persistOp(op: ObservedOp) { async function persistOp(op: ObservedOp) {
const csv = `timestamp,endpoint,input_summary,success,duration_ms,output_summary,error try {
"${op.timestamp}","${op.endpoint}","${esc(op.input_summary)}",${op.success},${op.duration_ms},"${esc(op.output_summary)}","${esc(op.error || "")}"`; const { mkdir, appendFile } = await import("node:fs/promises");
await mkdir("data/_observer", { recursive: true });
const form = new FormData(); await appendFile("data/_observer/ops.jsonl", JSON.stringify(op) + "\n");
form.append("file", new Blob([csv], { type: "text/csv" }), "op.csv"); } catch {
await fetch(`${LAKEHOUSE}/ingest/file?name=observed_operations`, { method: "POST", body: form }).catch(() => {}); // Persistence is best-effort; in-memory ring still works.
}
} }
function esc(s: string) { return s.replace(/"/g, '""').replace(/\n/g, ' '); }
// ─── Error analyzer loop ─── // ─── Error analyzer loop ───
@@ -189,10 +220,74 @@ async function consolidatePlaybooks() {
} }
} }
// ─── HTTP listener for external ops (Phase 24) ───
// Scenarios POST per-event outcomes here so the observer's analyzer +
// playbook builder see them alongside MCP-wrapped ops. Read-only stats
// also exposed at /stats for external health checks.
function startHttpListener() {
Bun.serve({
port: OBSERVER_PORT,
hostname: "0.0.0.0",
fetch(req) {
const url = new URL(req.url);
if (req.method === "GET" && url.pathname === "/health") {
return new Response(JSON.stringify({ status: "ok", ops_in_ring: recentOps.length }));
}
if (req.method === "GET" && url.pathname === "/stats") {
const bySource = new Map<string, number>();
for (const o of recentOps) {
const k = o.source ?? "mcp";
bySource.set(k, (bySource.get(k) ?? 0) + 1);
}
return new Response(JSON.stringify({
total: recentOps.length,
successes: recentOps.filter(o => o.success).length,
failures: recentOps.filter(o => !o.success).length,
by_source: Object.fromEntries(bySource),
recent_scenario_ops: recentOps
.filter(o => o.source === "scenario")
.slice(-10)
.map(o => ({ ts: o.timestamp, ok: o.success, staffer: o.staffer_id, kind: o.event_kind, role: o.role })),
}));
}
if (req.method === "POST" && url.pathname === "/event") {
return req.json().then((body: any) => {
const op: ObservedOp = {
timestamp: body.timestamp ?? new Date().toISOString(),
endpoint: body.endpoint ?? "scenario:fill",
input_summary: body.input_summary ?? `${body.event_kind ?? "?"} ${body.role ?? "?"}×${body.count ?? "?"} in ${body.city ?? "?"}, ${body.state ?? "?"}`,
success: !!body.success,
duration_ms: Number(body.duration_ms ?? 0),
output_summary: body.output_summary ?? (body.success ? "filled" : (body.error ?? "failed")),
error: body.error,
source: "scenario",
staffer_id: body.staffer_id,
sig_hash: body.sig_hash,
event_kind: body.event_kind,
role: body.role,
city: body.city,
state: body.state,
count: body.count,
rescue_attempted: !!body.rescue_attempted,
rescue_succeeded: !!body.rescue_succeeded,
};
recordExternalOp(op);
persistOp(op).catch(() => {});
return new Response(JSON.stringify({ accepted: true, ring_size: recentOps.length }));
}).catch((e: Error) =>
new Response(JSON.stringify({ error: e.message }), { status: 400 }));
}
return new Response("not found", { status: 404 });
},
});
console.error(`[observer] HTTP listener bound to 0.0.0.0:${OBSERVER_PORT}`);
}
// ─── Main loop ─── // ─── Main loop ───
async function main() { async function main() {
console.error(`[observer] started — cycle=${CYCLE_SECS}s, gateway=${GATEWAY}`); console.error(`[observer] started — cycle=${CYCLE_SECS}s, gateway=${GATEWAY}, port=${OBSERVER_PORT}`);
// Run a health check first // Run a health check first
const health = await fetch(`${GATEWAY}/health`).then(r => r.json()).catch(() => null); const health = await fetch(`${GATEWAY}/health`).then(r => r.json()).catch(() => null);
@@ -202,6 +297,9 @@ async function main() {
} }
console.error(`[observer] gateway healthy: ${JSON.stringify(health)}`); console.error(`[observer] gateway healthy: ${JSON.stringify(health)}`);
// Phase 24 — bind HTTP listener so scenarios can POST outcomes.
startHttpListener();
// Main loop // Main loop
let cycle = 0; let cycle = 0;
while (true) { while (true) {
@@ -216,11 +314,13 @@ async function main() {
await consolidatePlaybooks(); await consolidatePlaybooks();
} }
const scenarioOps = recentOps.filter(o => o.source === "scenario").length;
const stats = { const stats = {
cycle, cycle,
total_ops: recentOps.length, total_ops: recentOps.length,
successes: recentOps.filter(o => o.success).length, successes: recentOps.filter(o => o.success).length,
failures: recentOps.filter(o => !o.success).length, failures: recentOps.filter(o => !o.success).length,
scenario_ops: scenarioOps,
}; };
console.error(`[observer] cycle ${cycle}: ${JSON.stringify(stats)}`); console.error(`[observer] cycle ${cycle}: ${JSON.stringify(stats)}`);
} }
mcp-server/onboard.html
+414
View File
@@ -0,0 +1,414 @@
<!DOCTYPE html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>Lakehouse — Connect Your Data</title>
<style>
*{margin:0;padding:0;box-sizing:border-box}
body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;color:#b0b8c4;font-size:14px;line-height:1.55;-webkit-font-smoothing:antialiased}
a{color:#58a6ff;text-decoration:none}
a:hover{color:#79c0ff}
.bar{background:#0d1117;padding:0 24px;height:56px;border-bottom:1px solid #171d27;display:flex;justify-content:space-between;align-items:center;position:sticky;top:0;z-index:10}
.bar h1{font-size:14px;font-weight:600;color:#e6edf3;letter-spacing:-0.2px}
.bar nav{display:flex;gap:2px}
.bar nav a{font-size:12px;color:#545d68;padding:6px 14px;border-radius:6px;transition:all 0.15s}
.bar nav a:hover{color:#e6edf3;background:#161b22}
.bar nav a.active{color:#e6edf3;background:#1c2333}
.bar .rt{font-size:11px;color:#545d68}
.wrap{max-width:1040px;margin:0 auto;padding:28px 20px 60px}
.step{margin-bottom:36px;padding-left:44px;position:relative}
.step:before{content:attr(data-n);position:absolute;left:0;top:-2px;width:30px;height:30px;border-radius:15px;background:#0d1117;border:1px solid #21262d;color:#58a6ff;font-weight:700;font-size:13px;display:flex;align-items:center;justify-content:center}
.step.active:before{background:#1f6feb;color:#fff;border-color:#1f6feb}
.step.done:before{content:"✓";background:#0d2818;color:#3fb950;border-color:#2ea043}
.step h2{color:#e6edf3;font-size:18px;font-weight:700;margin-bottom:6px;letter-spacing:-0.2px}
.step .lede{color:#8b949e;font-size:13px;margin-bottom:14px;line-height:1.6}
.card{background:#0d1117;border:1px solid #171d27;border-radius:10px;padding:18px;margin:8px 0}
.btn{padding:10px 20px;background:#1f6feb;border:none;border-radius:8px;color:#fff;font-size:13px;font-weight:600;cursor:pointer;text-decoration:none;display:inline-block}
.btn:hover{background:#388bfd}
.btn:disabled{opacity:0.4;cursor:not-allowed}
.btn.ghost{background:transparent;border:1px solid #21262d;color:#c9d1d9}
.btn.ghost:hover{background:#161b22;border-color:#30363d}
.btn.green{background:#2ea043}
.btn.green:hover{background:#3fb950}
.drop{border:2px dashed #30363d;border-radius:10px;padding:28px;text-align:center;transition:all 0.15s;cursor:pointer}
.drop:hover{border-color:#58a6ff;background:#0d1117}
.drop.active{border-color:#2ea043;background:#0d2818}
.drop input[type=file]{display:none}
.drop .title{color:#e6edf3;font-weight:600;margin-bottom:4px}
.drop .sub{color:#545d68;font-size:12px}
.actions{display:flex;gap:10px;flex-wrap:wrap;margin-top:10px}
input[type=text]{padding:10px 14px;background:#161b22;border:1px solid #21262d;border-radius:8px;color:#e6edf3;font-size:13px;outline:none;width:100%}
input[type=text]:focus{border-color:#388bfd}
table.preview{width:100%;border-collapse:collapse;font-size:11px;margin-top:8px;font-family:ui-monospace,Menlo,monospace}
table.preview th{text-align:left;padding:8px 10px;color:#8b949e;background:#0d1117;border-bottom:1px solid #21262d;font-weight:600;font-size:10px;text-transform:uppercase;letter-spacing:0.8px}
table.preview td{padding:6px 10px;border-bottom:1px solid #171d27;color:#c9d1d9;white-space:nowrap;max-width:180px;overflow:hidden;text-overflow:ellipsis}
table.preview tr:hover td{background:#0d1117}
.col-list{display:flex;flex-direction:column;gap:6px;margin-top:6px}
.col{display:flex;align-items:center;gap:8px;padding:8px 12px;background:#161b22;border-radius:6px;font-size:12px}
.col .name{color:#e6edf3;font-weight:500;min-width:140px}
.col .type{color:#58a6ff;font-size:11px;font-family:ui-monospace,Menlo,monospace;min-width:60px}
.col .pii{padding:2px 8px;border-radius:9px;font-size:9px;font-weight:600;letter-spacing:0.3px}
.pii.email{background:#2d1b00;color:#d29922;border:1px solid #854d0e}
.pii.phone{background:#2d1b00;color:#d29922;border:1px solid #854d0e}
.pii.ssn{background:#3a1a1a;color:#f85149;border:1px solid #7f1d1d}
.pii.addr{background:#2d1b00;color:#d29922;border:1px solid #854d0e}
.col .sample{color:#8b949e;font-size:11px;font-family:ui-monospace,Menlo,monospace;flex:1;min-width:0;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}
.note{color:#8b949e;font-size:12px;line-height:1.7;padding:10px 14px;border-left:2px solid #21262d;margin:10px 0}
.note strong{color:#c9d1d9}
.result{padding:16px;border-radius:10px;margin:10px 0}
.result.ok{background:#0d2818;border:1px solid #2ea04360;color:#86efac}
.result.err{background:#3a1a1a;border:1px solid #f8514960;color:#fca5a5}
.result .big{font-size:16px;font-weight:700;margin-bottom:4px}
.result .detail{font-size:12px}
.spin{display:inline-block;width:14px;height:14px;border:2px solid #30363d;border-top-color:#58a6ff;border-radius:50%;animation:spin 0.7s linear infinite;vertical-align:middle;margin-right:6px}
@keyframes spin{to{transform:rotate(360deg)}}
.footer{border-top:1px solid #171d27;padding:20px;text-align:center;color:#3d444d;font-size:11px}
@media(max-width:720px){
.wrap{padding:20px 12px 40px}
.step{padding-left:36px}
.step:before{width:26px;height:26px;font-size:12px}
.bar nav{display:none}
}
</style></head>
<body>
<div class="bar">
<h1>Lakehouse — Connect Your Data</h1>
<nav>
<a href=".">Dashboard</a>
<a href="console">Walkthrough</a>
<a href="proof">Architecture</a>
<a href="spec">Spec</a>
<a href="onboard" class="active">Onboard</a>
<a href="alerts">Alerts</a>
<a href="workspaces">Workspaces</a>
</nav>
<div class="rt">30 minutes from CSV to live search</div>
</div>
<div class="wrap">
<div style="margin-bottom:24px">
<h1 style="color:#e6edf3;font-size:26px;font-weight:700;letter-spacing:-0.5px;margin-bottom:8px">Ingest any staffing CSV in three steps</h1>
<p style="color:#8b949e;font-size:14px;line-height:1.6;max-width:720px">
Upload your ATS export, your worker roster, or any CSV with a <code>name</code> column.
The wizard auto-detects columns, flags PII, previews the first rows, then ingests
as a queryable Parquet dataset. Everything that follows — hybrid search,
playbook ranking, pattern discovery — works against your data automatically.
</p>
</div>
<!-- Step 1 -->
<div class="step active" data-n="1" id="step-1">
<h2>Pick a file</h2>
<div class="lede">Drag a CSV in, pick from disk, or use the sample roster to see the flow without any real data.</div>
<label class="drop" id="drop">
<input type="file" id="file" accept=".csv,text/csv">
<div class="title">Drop a CSV here or click to choose</div>
<div class="sub">Max 50MB · UTF-8 · comma-separated with a header row</div>
</label>
<div class="actions">
<a class="btn ghost" href="samples/staffing_roster_sample.csv" download>Download sample roster (25 workers)</a>
<button class="btn ghost" id="use-sample" type="button">Use the sample as my data</button>
</div>
</div>
<!-- Step 2 -->
<div class="step" data-n="2" id="step-2" style="display:none">
<h2>Review what we detected</h2>
<div class="lede">
Columns auto-typed. PII columns flagged. First rows previewed. Nothing is written to the system yet — this is a read-only dry-run.
</div>
<div class="card">
<div style="color:#545d68;font-size:11px;text-transform:uppercase;letter-spacing:1.2px;font-weight:600;margin-bottom:10px">Columns detected</div>
<div class="col-list" id="col-list"></div>
</div>
<div class="card">
<div style="color:#545d68;font-size:11px;text-transform:uppercase;letter-spacing:1.2px;font-weight:600;margin-bottom:10px">First rows</div>
<div style="overflow-x:auto"><table class="preview" id="preview-table"></table></div>
<div style="color:#545d68;font-size:11px;margin-top:8px" id="row-count"></div>
</div>
<div class="note">
<strong>What happens next.</strong> On commit, the file is sent to <code>/ingest/file</code> — the same
endpoint every other ingest path uses. The Rust gateway writes it to object storage as
Parquet, computes a schema fingerprint, registers it in the catalog, and auto-detects PII
columns server-side. Re-uploading the same file is a no-op (deduplicated by content hash).
</div>
</div>
<!-- Step 3 -->
<div class="step" data-n="3" id="step-3" style="display:none">
<h2>Name it and commit</h2>
<div class="lede">Give the dataset a queryable name. This becomes the table you can <code>SELECT * FROM</code> immediately after commit.</div>
<div class="card">
<label style="display:block;color:#8b949e;font-size:12px;margin-bottom:6px">Dataset name</label>
<input type="text" id="dataset-name" placeholder="e.g. acme_staffing_roster" value="client_workers">
<div style="color:#545d68;font-size:11px;margin-top:6px">
Use lowercase + underscores. Once committed: queryable via <code>/query/sql</code>,
searchable via <code>/search</code>, indexable via <code>/vectors/index</code>.
</div>
<div class="actions" style="margin-top:14px">
<button class="btn green" id="commit-btn" type="button">Commit dataset</button>
<button class="btn ghost" id="back-btn" type="button">Back</button>
</div>
</div>
<div id="commit-result"></div>
</div>
<!-- Step 4: Post-commit next steps (hidden until success) -->
<div class="step" data-n="4" id="step-4" style="display:none">
<h2>Your dataset is live</h2>
<div class="lede">From here, the rest of the system applies to your data with zero additional setup.</div>
<div class="card">
<div style="color:#e6edf3;font-size:14px;font-weight:600;margin-bottom:12px">What you can do right now</div>
<div style="display:flex;flex-direction:column;gap:10px">
<div style="padding:10px 14px;background:#161b22;border-radius:6px;font-size:12px"><strong style="color:#e6edf3">Query via SQL.</strong> <code>POST /query/sql</code> with <code>SELECT * FROM <span id="ds-name-1">your_dataset</span> LIMIT 10</code>.</div>
<div style="padding:10px 14px;background:#161b22;border-radius:6px;font-size:12px"><strong style="color:#e6edf3">Build a vector index.</strong> <code>POST /vectors/index</code> with <code>{"dataset":"<span id="ds-name-2">your_dataset</span>","text_column":"skills"}</code>. Embeddings stream in; queryable progressively.</div>
<div style="padding:10px 14px;background:#161b22;border-radius:6px;font-size:12px"><strong style="color:#e6edf3">Search via the dashboard.</strong> Open the <a href=".">dashboard</a> and the "Search all workers" box. Results will come from your data.</div>
<div style="padding:10px 14px;background:#161b22;border-radius:6px;font-size:12px"><strong style="color:#e6edf3">Track with playbook memory.</strong> Every Call/SMS/No-show click on a worker card trains the system on your data.</div>
</div>
</div>
</div>
</div>
<div class="footer">Lakehouse · onboarding wizard · <a href="spec">full architecture spec</a></div>
<script>
var P=location.pathname.indexOf('/lakehouse')>=0?'/lakehouse':'';
var A=location.origin+P;
var parsedFile=null; // { name, rows, columns, sample }
var rawBlob=null;
function el(tag,cls,text){
var e=document.createElement(tag);
if(cls) e.className=cls;
if(text!==undefined && text!==null) e.textContent=String(text);
return e;
}
function setStep(n){
for(var i=1;i<=4;i++){
var s=document.getElementById('step-'+i);
if(!s) continue;
s.classList.remove('active','done');
if(i<n) s.classList.add('done');
if(i===n) { s.classList.add('active'); s.style.display='block'; }
else if(i>n) s.style.display='none';
else s.style.display='block';
}
}
// Basic CSV parser. Handles quoted fields with commas + escaped quotes.
function parseCSV(text){
var lines=[], i=0, cur='', row=[], inQ=false;
while(i<text.length){
var c=text[i];
if(inQ){
if(c==='"' && text[i+1]==='"'){ cur+='"'; i+=2; continue; }
if(c==='"'){ inQ=false; i++; continue; }
cur+=c; i++;
} else {
if(c==='"'){ inQ=true; i++; continue; }
if(c===','){ row.push(cur); cur=''; i++; continue; }
if(c==='\n'){ row.push(cur); lines.push(row); row=[]; cur=''; i++; continue; }
if(c==='\r'){ i++; continue; }
cur+=c; i++;
}
}
if(cur.length>0||row.length>0){ row.push(cur); lines.push(row); }
return lines;
}
// Type + PII inference from a sample of values for a column.
function inferColumn(name, values){
var nonEmpty=values.filter(function(v){return v!=null && v!==''});
var n=nonEmpty.length;
var lname=(name||'').toLowerCase();
// PII signals by name and sample content
var pii=null;
if(/email|e-mail|mail_address/.test(lname)) pii='email';
else if(/phone|mobile|cell|tel/.test(lname)) pii='phone';
else if(/ssn|social|tax_id/.test(lname)) pii='ssn';
else if(/address|street|zip|postal/.test(lname)) pii='addr';
// Content-based PII scan on sample
if(!pii){
var emailRe=/^[^\s@]+@[^\s@]+\.[^\s@]+$/;
var phoneRe=/^[\s\-\(\)\+]*\d[\s\-\(\)\+\d]{6,}$/;
var emailHits=nonEmpty.filter(function(v){return emailRe.test(v)}).length;
var phoneHits=nonEmpty.filter(function(v){return phoneRe.test(v)}).length;
if(emailHits>n*0.5) pii='email';
else if(phoneHits>n*0.5) pii='phone';
}
// Type inference
var type='text';
if(n>0){
var intHits=nonEmpty.filter(function(v){return /^-?\d+$/.test(v)}).length;
var numHits=nonEmpty.filter(function(v){return /^-?\d+(\.\d+)?$/.test(v)}).length;
if(intHits===n) type='int';
else if(numHits===n) type='decimal';
else if(nonEmpty.every(function(v){return /^\d{4}-\d{2}-\d{2}/.test(v)})) type='date';
}
return { name: name, type: type, pii: pii, sample: nonEmpty.slice(0,3).join(' · ') || '(empty)' };
}
function readFile(file){
rawBlob=file;
var reader=new FileReader();
reader.onload=function(e){
var text=String(e.target.result||'');
var lines=parseCSV(text);
if(lines.length<2){
alert('CSV is empty or has no data rows.');
return;
}
var header=lines[0].map(function(h){return String(h||'').trim()});
var rows=lines.slice(1).filter(function(r){return r.length>0 && r.some(function(c){return c!=null&&c!==''})});
// Build columns with inference
var columns=header.map(function(h,idx){
var colValues=rows.map(function(r){return r[idx]});
return inferColumn(h, colValues.slice(0,50)); // sample first 50 for perf
});
parsedFile={name: file.name, rowCount: rows.length, header: header, rows: rows, columns: columns};
renderPreview();
setStep(2);
};
reader.readAsText(file);
}
function renderPreview(){
var colList=document.getElementById('col-list'); colList.textContent='';
parsedFile.columns.forEach(function(c){
var row=el('div','col');
row.appendChild(el('span','name',c.name));
row.appendChild(el('span','type',c.type));
if(c.pii){
var chip=el('span','pii '+c.pii,c.pii.toUpperCase());
row.appendChild(chip);
}
row.appendChild(el('span','sample',c.sample));
colList.appendChild(row);
});
var table=document.getElementById('preview-table');table.textContent='';
var thead=document.createElement('thead');
var htr=document.createElement('tr');
parsedFile.header.forEach(function(h){ htr.appendChild(el('th',null,h)) });
thead.appendChild(htr);table.appendChild(thead);
var tbody=document.createElement('tbody');
parsedFile.rows.slice(0,5).forEach(function(r){
var tr=document.createElement('tr');
parsedFile.header.forEach(function(_,idx){
tr.appendChild(el('td',null,r[idx]||''));
});
tbody.appendChild(tr);
});
table.appendChild(tbody);
document.getElementById('row-count').textContent=parsedFile.rowCount.toLocaleString()+' data rows · '+parsedFile.header.length+' columns · source: '+parsedFile.name;
// Auto-advance button: a Next button at the bottom of step-2
var step2=document.getElementById('step-2');
var existing=document.getElementById('step-2-next');
if(!existing){
var wrap=el('div','actions');wrap.id='step-2-next';
var next=el('button','btn','Continue to commit');next.type='button';
next.onclick=function(){ setStep(3); };
var reset=el('button','btn ghost','Pick a different file');reset.type='button';
reset.onclick=function(){ parsedFile=null; rawBlob=null; document.getElementById('file').value=''; setStep(1); };
wrap.appendChild(next);wrap.appendChild(reset);
step2.appendChild(wrap);
}
}
function commit(){
var dsName=(document.getElementById('dataset-name').value||'').trim();
if(!dsName){ alert('Dataset name required.'); return; }
if(!/^[a-z][a-z0-9_]*$/.test(dsName)){ alert('Dataset name: lowercase letters, numbers, underscores only; start with a letter.'); return; }
if(!rawBlob){ alert('No file loaded.'); return; }
var btn=document.getElementById('commit-btn');
btn.disabled=true;btn.textContent='';
var spin=el('span','spin');btn.appendChild(spin);
btn.appendChild(document.createTextNode('Ingesting…'));
var form=new FormData();
form.append('file', rawBlob, parsedFile.name);
fetch(A+'/onboard/ingest?name='+encodeURIComponent(dsName), {
method: 'POST', body: form,
}).then(function(r){
return r.text().then(function(body){ return {status:r.status, body:body}; });
}).then(function(res){
btn.disabled=false;btn.textContent='Commit dataset';
var out=document.getElementById('commit-result');out.textContent='';
if(res.status>=200 && res.status<300){
var ok=el('div','result ok');
ok.appendChild(el('div','big','Dataset "'+dsName+'" is live.'));
ok.appendChild(el('div','detail',parsedFile.rowCount.toLocaleString()+' rows · '+parsedFile.header.length+' columns · registered in catalog'));
out.appendChild(ok);
document.getElementById('ds-name-1').textContent=dsName;
document.getElementById('ds-name-2').textContent=dsName;
setStep(4);
} else {
var err=el('div','result err');
err.appendChild(el('div','big','Ingest failed ('+res.status+')'));
err.appendChild(el('div','detail',res.body.slice(0,400)));
out.appendChild(err);
}
}).catch(function(e){
btn.disabled=false;btn.textContent='Commit dataset';
var out=document.getElementById('commit-result');out.textContent='';
var err=el('div','result err');
err.appendChild(el('div','big','Network error'));
err.appendChild(el('div','detail',e.message||String(e)));
out.appendChild(err);
});
}
document.addEventListener('DOMContentLoaded',function(){
var fileInput=document.getElementById('file');
var drop=document.getElementById('drop');
fileInput.addEventListener('change',function(e){
if(e.target.files.length) readFile(e.target.files[0]);
});
drop.addEventListener('dragover',function(e){e.preventDefault();drop.classList.add('active')});
drop.addEventListener('dragleave',function(){drop.classList.remove('active')});
drop.addEventListener('drop',function(e){
e.preventDefault();drop.classList.remove('active');
if(e.dataTransfer.files.length) readFile(e.dataTransfer.files[0]);
});
document.getElementById('use-sample').addEventListener('click',function(){
// Fetch the sample CSV and feed it through the same pipeline
fetch(A+'/samples/staffing_roster_sample.csv').then(function(r){return r.blob()}).then(function(blob){
var file=new File([blob],'staffing_roster_sample.csv',{type:'text/csv'});
readFile(file);
}).catch(function(e){ alert('Sample unavailable: '+e.message); });
});
document.getElementById('commit-btn').addEventListener('click',commit);
document.getElementById('back-btn').addEventListener('click',function(){ setStep(2); });
});
</script>
</body></html>
mcp-server/proof.html
+457
View File
@@ -0,0 +1,457 @@
<!DOCTYPE html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>Lakehouse — Architecture &amp; Reproduction</title>
<style>
*{margin:0;padding:0;box-sizing:border-box}
body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;color:#b0b8c4;font-size:14px;line-height:1.55;-webkit-font-smoothing:antialiased}
a{color:#58a6ff;text-decoration:none}
a:hover{color:#79c0ff}
.bar{background:#0d1117;padding:0 24px;height:56px;border-bottom:1px solid #171d27;display:flex;justify-content:space-between;align-items:center;position:sticky;top:0;z-index:10}
.bar h1{font-size:14px;font-weight:600;color:#e6edf3;letter-spacing:-0.2px}
.bar nav{display:flex;gap:2px}
.bar nav a{font-size:12px;color:#545d68;padding:6px 14px;border-radius:6px;transition:all 0.15s}
.bar nav a:hover{color:#e6edf3;background:#161b22}
.bar nav a.active{color:#e6edf3;background:#1c2333}
.bar .rt{font-size:11px;color:#545d68}
.wrap{max-width:1040px;margin:0 auto;padding:28px 20px 60px}
.chapter{margin-bottom:48px}
.chapter .num{color:#545d68;font-size:11px;font-weight:600;letter-spacing:1.6px;text-transform:uppercase;margin-bottom:6px}
.chapter h2{color:#e6edf3;font-size:24px;font-weight:700;letter-spacing:-0.4px;margin-bottom:8px;line-height:1.2}
.chapter .lede{color:#8b949e;font-size:14px;margin-bottom:18px;max-width:680px;line-height:1.6}
.chapter h3{color:#e6edf3;font-size:16px;font-weight:600;margin:18px 0 6px}
.card{background:#0d1117;border:1px solid #171d27;border-radius:12px;padding:20px;margin-bottom:12px}
.grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(220px,1fr));gap:10px}
.stat-lg{padding:18px 20px}
.stat-lg .n{font-size:26px;font-weight:800;color:#e6edf3;letter-spacing:-1px;line-height:1}
.stat-lg .l{font-size:10px;color:#545d68;text-transform:uppercase;letter-spacing:1.2px;margin-top:8px;font-weight:600}
.stat-lg .sub{font-size:12px;color:#8b949e;margin-top:4px}
.accent-l{border-left:3px solid #2ea043}
.accent-b{border-left:3px solid #1f6feb}
.accent-a{border-left:3px solid #bc8cff}
.accent-w{border-left:3px solid #d29922}
.accent-r{border-left:3px solid #f85149}
.row{display:flex;justify-content:space-between;align-items:center;gap:12px;padding:10px 14px;background:#0d1117;border:1px solid #171d27;border-radius:8px;margin-bottom:6px;font-size:13px}
.row:hover{border-color:#21262d}
.row .title{color:#e6edf3;font-weight:500}
.row .meta{color:#8b949e;font-size:11px;margin-top:2px;font-family:ui-monospace,Menlo,monospace}
.row .val{color:#58a6ff;font-weight:600;white-space:nowrap;font-family:ui-monospace,Menlo,monospace}
.row.pass{border-left:3px solid #2ea043}
.row.fail{border-left:3px solid #f85149}
table.plain{width:100%;border-collapse:collapse;font-size:12px;margin-top:8px}
table.plain th{text-align:left;padding:8px 12px;color:#545d68;font-weight:600;text-transform:uppercase;font-size:10px;letter-spacing:0.8px;border-bottom:1px solid #171d27}
table.plain td{padding:8px 12px;border-bottom:1px solid #171d27;color:#c9d1d9;font-family:ui-monospace,Menlo,monospace}
table.plain tr:hover td{background:#0d1117}
code{background:#161b22;color:#e6edf3;padding:2px 6px;border-radius:4px;font-family:ui-monospace,Menlo,monospace;font-size:12px}
pre{background:#161b22;border:1px solid #171d27;border-radius:8px;padding:14px 16px;overflow-x:auto;font-family:ui-monospace,Menlo,monospace;font-size:12px;color:#c9d1d9;line-height:1.5;margin:8px 0}
.narr{color:#8b949e;font-size:13px;line-height:1.7;margin:10px 0;padding:10px 14px;border-left:2px solid #21262d}
.narr strong{color:#c9d1d9;font-weight:600}
.ref{color:#545d68;font-size:11px;margin-top:6px;font-family:ui-monospace,Menlo,monospace}
.ref strong{color:#79c0ff;font-weight:600}
.math{background:#0d1117;border:1px solid #171d27;border-radius:8px;padding:14px 16px;font-family:ui-monospace,Menlo,monospace;font-size:13px;color:#e6edf3;margin:8px 0}
.math .c{color:#8b949e}
.footer{border-top:1px solid #171d27;padding:20px;text-align:center;color:#3d444d;font-size:11px}
.loading{color:#484f58;font-style:italic;padding:20px 0;text-align:center}
.err{color:#f85149;font-size:12px;padding:10px}
@media(max-width:720px){
.wrap{padding:20px 12px 40px}
.chapter h2{font-size:20px}
.bar nav{display:none}
}
</style></head>
<body>
<div class="bar">
<h1>Lakehouse — Architecture &amp; Reproduction</h1>
<nav>
<a href=".">Dashboard</a>
<a href="console">Walkthrough</a>
<a href="proof" class="active">Architecture</a>
<a href="spec">Spec</a>
<a href="onboard">Onboard</a>
<a href="alerts">Alerts</a>
<a href="workspaces">Workspaces</a>
</nav>
<div class="rt" id="hdr-time">Running live tests…</div>
</div>
<div class="wrap">
<div class="chapter">
<div class="num">Chapter 1</div>
<h2>Receipts, not promises</h2>
<div class="lede">Every test below ran live against the real gateway when you loaded this page. Sub-100ms SQL on multi-million-row Parquet, hybrid search with playbook boost applied. No fixtures. If a test fails, you'll see ✗.</div>
<div id="ch1-tests"><div class="loading">Running tests…</div></div>
</div>
<div class="chapter">
<div class="num">Chapter 2</div>
<h2>Architecture — 13 crates, one object store, one local AI runtime</h2>
<div class="lede">Request flows top to bottom. Every node is independently swappable. Every line is a real HTTP or gRPC hop that you can trace with <code>tcpdump</code>.</div>
<div class="card accent-b">
<pre> HTTP :3100 + gRPC :3101
│
┌───────▼───────┐
│ gateway │ Rust · Axum · routing, CORS, auth, tools
└───────┬───────┘
┌────────────┬───────────┼───────────┬────────────┐
│ │ │ │ │
┌────▼───┐ ┌────▼───┐ ┌────▼───┐ ┌────▼───┐ ┌────▼───┐
│catalog │ │ query │ │ vector │ │ ingest │ │aibridge│
│ d │ │ d │ │ d │ │ d │ │ │
└────┬───┘ └────┬───┘ └────┬───┘ └────┬───┘ └────┬───┘
│ │ │ │ │
└────────────┴───────────┼───────────┴────────────┘
▼
┌─────────────────┐
│ object storage │ Parquet files (local / S3)
└─────────────────┘
▲
│
┌───────┴────────┐
│ Python sidecar │ FastAPI → Ollama
│ (aibridge) │ local models only
└────────────────┘</pre>
</div>
<h3>Per-crate responsibility</h3>
<table class="plain">
<thead><tr><th>Crate</th><th>Role</th><th>Path</th></tr></thead>
<tbody>
<tr><td>shared</td><td>Types, errors, Arrow helpers, PII detection, secrets provider</td><td>crates/shared/</td></tr>
<tr><td>storaged</td><td>object_store I/O, BucketRegistry (multi-bucket), AppendLog, ErrorJournal</td><td>crates/storaged/</td></tr>
<tr><td>catalogd</td><td>Metadata authority — manifests, views, tombstones, profiles, schema fingerprints</td><td>crates/catalogd/</td></tr>
<tr><td>queryd</td><td>DataFusion SQL engine, MemTable cache, delta merge-on-read, compaction</td><td>crates/queryd/</td></tr>
<tr><td>ingestd</td><td>CSV/JSON/PDF(+OCR)/Postgres/MySQL ingest, cron schedules, auto-PII</td><td>crates/ingestd/</td></tr>
<tr><td>vectord</td><td>Embeddings as Parquet, HNSW, trial system, autotune agent, playbook_memory</td><td>crates/vectord/</td></tr>
<tr><td>vectord-lance</td><td>Firewall crate — Lance 4.0 + Arrow 57 isolated from main Arrow 55</td><td>crates/vectord-lance/</td></tr>
<tr><td>journald</td><td>Append-only mutation event log for time-travel &amp; audit</td><td>crates/journald/</td></tr>
<tr><td>aibridge</td><td>Rust↔Python sidecar, Ollama HTTP client, VRAM introspection</td><td>crates/aibridge/</td></tr>
<tr><td>gateway</td><td>Axum HTTP :3100 + gRPC :3101, middleware, tools registry</td><td>crates/gateway/</td></tr>
<tr><td>ui</td><td>Dioxus WASM internal developer UI</td><td>crates/ui/</td></tr>
<tr><td>mcp-server</td><td>Bun TypeScript recruiter-facing app (this server)</td><td>mcp-server/</td></tr>
</tbody>
</table>
<div class="ref"><strong>Source:</strong> git.agentview.dev/profit/lakehouse &nbsp;·&nbsp; <strong>ADRs:</strong> docs/DECISIONS.md (currently 20 records)</div>
</div>
<div class="chapter">
<div class="num">Chapter 3</div>
<h2>Dual-agent recursive consensus loop</h2>
<div class="lede">The system we use to execute staffing fills is a dual-agent recursive protocol. Two agents with distinct roles iterate against a shared log until one of three terminal states is reached. It is deterministic in structure, stochastic in content, and verifiable through the per-run log artifact.</div>
<h3>Agents and protocol</h3>
<div class="card accent-a">
<pre> task in
│
▼
┌───────────────────────────────────────────────────────────┐
│ EXECUTOR (mistral:latest) │
│ ──────────────────────────────────────────────────────── │
│ input: task spec + shared log + seen-candidates ledger │
│ output: one JSON action per turn │
│ · {kind:"plan",steps:[…]} │
│ · {kind:"tool_call",tool,args,rationale} │
│ · {kind:"propose_done",fills:[N of N]} │
└───────────┬───────────────────────────────┬───────────────┘
│ tool_call │ propose_done
▼ │
┌──────────────────────────┐ │
│ TOOL DISPATCH │ │
│ hybrid_search / sql │ │
│ (against live gateway) │ │
└──────────┬───────────────┘ │
│ result (trimmed, exclusions) │
▼ ▼
┌───────────────────────────────────────────────────────────┐
│ REVIEWER (qwen2.5:latest) │
│ ──────────────────────────────────────────────────────── │
│ input: task spec + shared log (including tool result) │
│ output: {kind:"critique",verdict:"continue|drift| │
│ approve_done",notes} │
└───────────┬───────────────────────────────────────────────┘
│
┌─────┴─────┐
▼ ▼ ▼
continue drift approve_done + propose_done ⟹ SEAL
(next turn) (cap ≈ 3 →
hard abort)
</pre>
</div>
<div class="ref"><strong>Code:</strong> tests/multi-agent/agent.ts (protocol + prompts) &nbsp;·&nbsp; tests/multi-agent/orchestrator.ts (run loop) &nbsp;·&nbsp; tests/multi-agent/scenario.ts (5-event warehouse week)</div>
<h3>Why "dual" — role specialization</h3>
<div class="narr">
<strong>The executor is an optimist.</strong> Its job is to produce progress: pull candidates, verify SQL, propose consensus. It's instructed to be decisive.
<br><br>
<strong>The reviewer is a pessimist.</strong> Its job is to catch drift: proposals that don't match the task's geography, fill count, or role. It's authorized to stop the loop.
<br><br>
This adversarial separation is cheaper and more deterministic than asking a single model to self-critique. The reviewer has a hard rule: on the turn after a <code>propose_done</code>, it MUST emit either <code>approve_done</code> or <code>drift</code> — it cannot stall with <code>continue</code>.
</div>
<h3>Why "parallel" — orchestrator can fan out</h3>
<div class="narr">
<strong>Independent pairs run concurrently.</strong> <code>tests/multi-agent/run_e2e_rated.ts</code> runs two task-specific agent pairs via <code>Promise.all</code>. Ollama serializes inference at the model level, so "parallel" is concurrent orchestration — but the substrate (gateway, queryd, vectord) handles concurrent requests cleanly. Verified in the scenario harness: two contracts sealing simultaneously.
</div>
<h3>Why "recursive" — each seal feeds the next</h3>
<div class="narr">
<strong>Consensus does not end at the sealed playbook.</strong> Every sealed playbook is persisted to <code>playbook_memory</code> via <code>POST /vectors/playbook_memory/seed</code>. The next hybrid search for a semantically similar operation consults that memory via <code>compute_boost_for(query_embedding, top_k, base_weight)</code> and re-ranks the candidate pool. The system builds on itself turn over turn, playbook over playbook.
</div>
<h3>Termination guarantees</h3>
<div class="math">
<span class="c">// three paths out, every run has one of these:</span><br>
sealed = executor.propose_done ∧ reviewer.approve_done ∧ fills.count == target<br>
abort = consecutive_tool_errors ≥ MAX_TOOL_ERRORS (3) &nbsp;&nbsp;<span class="c">// executor can't form a valid call</span><br>
abort = consecutive_drifts ≥ MAX_CONSECUTIVE_DRIFTS (3) &nbsp;<span class="c">// reviewer keeps flagging</span><br>
abort = turn &gt; MAX_TURNS (12) &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="c">// no consensus reached in window</span>
</div>
<div class="narr">Every abort dumps the full log to <code>tests/multi-agent/playbooks/&lt;id&gt;-FAILED.json</code> for forensic review. No consensus is ever implicit.</div>
</div>
<div class="chapter">
<div class="num">Chapter 4</div>
<h2>Playbook memory — the compounding feedback loop</h2>
<div class="lede">A CRM stores events. This system turns events into re-ranking signal. Every sealed playbook endorses specific (worker, city, state) tuples. Every failure penalizes them. Every similar future query inherits the signal through cosine similarity.</div>
<h3>Seed shape</h3>
<div class="math">
PlaybookEntry {<br>
&nbsp;&nbsp;playbook_id, <span class="c">// pb-seed-&lt;sha8&gt;</span><br>
&nbsp;&nbsp;operation, <span class="c">// "fill: Welder x2 in Toledo, OH"</span><br>
&nbsp;&nbsp;approach, context, <span class="c">// short canonical — long strings dilute embedding</span><br>
&nbsp;&nbsp;timestamp, <span class="c">// RFC3339</span><br>
&nbsp;&nbsp;endorsed_names[], <span class="c">// validated against workers_500k for city+state</span><br>
&nbsp;&nbsp;city, state, <span class="c">// parsed from operation</span><br>
&nbsp;&nbsp;embedding <span class="c">// 768-d nomic-embed-text of text shape</span><br>
}
</div>
<div class="ref"><strong>Code:</strong> crates/vectord/src/playbook_memory.rs (PlaybookEntry, FailureRecord, PlaybookMemoryState)</div>
<h3>Boost math (positive + decay + negative)</h3>
<div class="math">
<span class="c">// For each playbook pb among top-K most cosine-similar:</span><br>
<span class="c">// given query embedding qv, constant base_weight, n_workers = |pb.endorsed_names|</span><br>
<br>
similarity = cosine(qv, pb.embedding) &nbsp;&nbsp; <span class="c">// skip if ≤ 0.05</span><br>
age_days = (now - pb.timestamp) / 86_400 seconds<br>
decay = e<sup>-age_days / 30</sup> &nbsp;&nbsp;<span class="c">// half-life = 30 days</span><br>
<br>
<span class="c">// For each endorsed worker in pb:</span><br>
key = (pb.city, pb.state, name)<br>
fail_count = failures[key] &nbsp;&nbsp;<span class="c">// # times this worker was marked no-show for same geo</span><br>
penalty = 0.5<sup>min(fail_count, 20)</sup><br>
<br>
per_worker = similarity × base_weight × decay × penalty / n_workers<br>
boost[key] = min(boost[key] + per_worker, MAX_BOOST_PER_WORKER)<br>
<br>
<span class="c">// MAX_BOOST_PER_WORKER = 0.25 — cap stops one popular worker from always winning</span>
</div>
<div class="ref"><strong>Code:</strong> crates/vectord/src/playbook_memory.rs::compute_boost_for &nbsp;·&nbsp; constants: MAX_BOOST_PER_WORKER, DEFAULT_TOP_K_PLAYBOOKS, BOOST_HALF_LIFE_DAYS</div>
<h3>Application at query time</h3>
<div class="math">
<span class="c">// In /vectors/hybrid handler (crates/vectord/src/service.rs):</span><br>
1. SQL filter narrows workers_500k to geo/role/availability<br>
2. Vector index returns top_k × 5 candidates by cosine to question<br>
3. compute_boost_for(qv, k=200) returns boost map<br>
4. For each candidate: parse (name, city, state) from chunk, look up boost, add to score<br>
5. Re-sort sources by boosted score<br>
6. Truncate to requested top_k, return with playbook_boost and playbook_citations
</div>
<div class="narr">
<strong>Why k=200.</strong> Direct measurement showed cosine similarity clusters in the 0.55-0.67 band across all playbooks regardless of geo (nomic-embed-text has narrow discrimination on this kind of structured operation text). A k of 25 silently missed geo-matched playbooks. k=200 is the measured floor for reliably catching compounding. Brute-force over 200 × 768-d is sub-ms even on this hardware.
</div>
<div class="ref"><strong>Evidence:</strong> Chicago Electrician compounding test 2026-04-20 — Carmen Green, Anna Patel, Fatima Wilson went from rank &gt;5 / boost 0 / 0 citations (run 0, no seed) to rank 1/2/3 / boost +0.250 (capped) / 3 citations each (run 3, after 3 identical seeds). Each seed increments citations; total boost caps at 0.25/worker.</div>
<h3>Write-through to SQL</h3>
<div class="narr">
<strong>successful_playbooks_live</strong> is a DataFusion-queryable Parquet surface maintained by <code>POST /vectors/playbook_memory/persist_sql</code>. Every <code>/log</code> from the recruiter UI triggers seed → persist_sql. The in-memory store and the SQL surface stay synchronized (full snapshot on each persist, safe because memory is source of truth).
</div>
<div class="ref"><strong>Code:</strong> crates/vectord/src/playbook_memory.rs::persist_to_sql &nbsp;·&nbsp; catalog-registered under "successful_playbooks_live"</div>
<h3>Pattern discovery (Path 2 — meta-index)</h3>
<div class="narr">
<strong>Beyond "who was endorsed."</strong> <code>POST /vectors/playbook_memory/patterns</code> takes a query, finds top-K similar past playbooks, pulls each endorsed worker's full workers_500k profile, and aggregates shared traits: recurring certifications, skill frequencies, modal archetype, reliability distribution. Returns a <code>discovered_pattern</code> string showing operator-actionable signal the user didn't explicitly query for.
</div>
<div class="ref"><strong>Code:</strong> crates/vectord/src/playbook_memory.rs::discover_patterns &nbsp;·&nbsp; <strong>Surfaces:</strong> /vectors/playbook_memory/patterns endpoint, /intelligence/chat response, /intelligence/permit_contracts cards</div>
</div>
<div class="chapter">
<div class="num">Chapter 5</div>
<h2>Key architectural choices — what was picked and why</h2>
<div class="lede">Each choice is documented in <code>docs/DECISIONS.md</code> (Architecture Decision Records). If you dispute any of these, the ADR names the alternatives we rejected and the measurement that drove the call.</div>
<div class="card">
<div class="row accent-b">
<div style="flex:1"><div class="title">ADR-001 · Object storage as source of truth</div><div class="meta">No traditional database. All data is Parquet on S3-compatible object storage. Eliminates DB operational overhead; every engine can read Parquet.</div></div>
</div>
<div class="row accent-b">
<div style="flex:1"><div class="title">ADR-008 · Embeddings stored as Parquet, not a vector DB</div><div class="meta">Keeps all data in one portable format. No Pinecone/Weaviate/Qdrant lock-in. Trade-off: brute-force search up to ~100K; HNSW beyond.</div></div>
</div>
<div class="row accent-l">
<div style="flex:1"><div class="title">ADR-012 · Append-only event journal — never destroy evidence</div><div class="meta">Every mutation is appended. Compliance, audit, AI-decision forensics. Impossible to retrofit; easy to add now.</div></div>
</div>
<div class="row accent-a">
<div style="flex:1"><div class="title">ADR-015 · Tool registry before raw SQL for agents</div><div class="meta">Named, governed, audited actions for agents. Permission checks, rate limits, parameter validation. MCP-compatible.</div></div>
</div>
<div class="row accent-w">
<div style="flex:1"><div class="title">ADR-019 · Hybrid Parquet+HNSW ⊕ Lance vector backend</div><div class="meta">Parquet+HNSW primary (2.55× faster search at 100K). Lance secondary for index-build speed (14× faster), random fetch (112× faster), append (structural). Per-profile <code>vector_backend: Parquet | Lance</code>.</div></div>
</div>
<div class="row accent-r">
<div style="flex:1"><div class="title">ADR-020 · Idempotent register() with schema-fingerprint gate</div><div class="meta">Same (name, fingerprint) reuses manifest. Different fingerprint = 409 Conflict. Prevents silent duplicate manifests. Cleanup run collapsed 374 → 31 datasets.</div></div>
</div>
<div class="row accent-l">
<div style="flex:1"><div class="title">Phase 19 design note · Statistical + semantic, not neural</div><div class="meta">Meta-index is cosine similarity + endorsement aggregation. No model training. Rebuildable from <code>successful_playbooks</code> alone. Neural re-ranker deferred to Phase 20+ only if statistical floor plateaus.</div></div>
</div>
</div>
</div>
<div class="chapter">
<div class="num">Chapter 6</div>
<h2>Measured at scale, on this machine</h2>
<div class="lede">Hardware: i9 + 128GB RAM + Nvidia A4000 16GB VRAM. Numbers below are from <em>this</em> running instance. Refresh the page and they'll recompute.</div>
<div class="grid" id="ch6-scale"><div class="loading">Loading scale data…</div></div>
<div id="ch6-recall" style="margin-top:10px"></div>
</div>
<div class="chapter">
<div class="num">Chapter 7</div>
<h2>Verify or dispute — reproduce it yourself</h2>
<div class="lede">Every claim below is a curl away from falsification.</div>
<div class="card">
<div class="narr"><strong>Health.</strong> Should return <code>lakehouse ok</code>.</div>
<pre>curl http://localhost:3100/health</pre>
<div class="narr"><strong>Any SQL on multi-million-row Parquet.</strong> Sub-100ms typical.</div>
<pre>curl -s -X POST http://localhost:3100/query/sql \
-H 'Content-Type: application/json' \
-d '{"sql":"SELECT role, COUNT(*) FROM workers_500k WHERE state=\"IL\" GROUP BY role LIMIT 5"}'</pre>
<div class="narr"><strong>Hybrid search with playbook boost.</strong> The whole Phase 19 feedback loop in one request.</div>
<pre>curl -s -X POST http://localhost:3100/vectors/hybrid \
-H 'Content-Type: application/json' \
-d '{"index_name":"workers_500k_v1",
"sql_filter":"role = '\''Forklift Operator'\'' AND city = '\''Chicago'\'' AND CAST(availability AS DOUBLE) > 0.5",
"question":"reliable forklift operator",
"top_k":5,"use_playbook_memory":true,"playbook_memory_k":200}'</pre>
<div class="narr"><strong>Playbook memory stats.</strong> Count + endorsed names + sample.</div>
<pre>curl http://localhost:3100/vectors/playbook_memory/stats</pre>
<div class="narr"><strong>Pattern discovery.</strong> What do past similar fills have in common?</div>
<pre>curl -s -X POST http://localhost:3100/vectors/playbook_memory/patterns \
-H 'Content-Type: application/json' \
-d '{"query":"Forklift Operator in Chicago, IL","top_k_playbooks":25,"min_trait_frequency":0.3}'</pre>
<div class="narr"><strong>Run the dual-agent scenario yourself.</strong> All 5 events, real fills, real artifacts.</div>
<pre>cd /home/profit/lakehouse
bun run tests/multi-agent/scenario.ts
# Output: tests/multi-agent/playbooks/scenario-&lt;timestamp&gt;/report.md</pre>
</div>
</div>
<div class="chapter">
<div class="num">Chapter 8</div>
<h2>What we are <em>not</em> claiming</h2>
<div class="lede">Every impressive-sounding number comes with a footnote. Here are the honest limits.</div>
<div class="card">
<div class="row accent-a"><div style="flex:1"><div class="title">workers_500k is synthetic.</div><div class="meta">Real client ATS export replaces this table. Schema is deliberately identical to a production ATS.</div></div></div>
<div class="row accent-a"><div style="flex:1"><div class="title">candidates table has 1,000 rows.</div><div class="meta">Intentionally small for demo. call_log references higher candidate_ids that don't cross-reference — this is a dataset alignment issue, not a pipeline issue.</div></div></div>
<div class="row accent-b"><div style="flex:1"><div class="title">Chicago permit data is real.</div><div class="meta">Pulled live from data.cityofchicago.org/resource/ydr8-5enu.json (Socrata API). Not synthetic. Not cached.</div></div></div>
<div class="row accent-l"><div style="flex:1"><div class="title">Playbook memory is seeded from demo runs.</div><div class="meta">The pipeline that seeds it is identical to what a live recruiter would trigger via /log. Same code path.</div></div></div>
<div class="row accent-w"><div style="flex:1"><div class="title">Local 7B models (mistral, qwen2.5) are imperfect.</div><div class="meta">They occasionally malform tool calls or drop fields. Multi-agent scenarios seal roughly 40-80% in one run. Larger models or constrained decoding would improve this. Not a substrate problem.</div></div></div>
<div class="row accent-r"><div style="flex:1"><div class="title">No rate/margin awareness yet.</div><div class="meta">Worker pay expectations vs contract bill rates are not modeled. Flagged as a Phase 20 item; no architectural blocker.</div></div></div>
</div>
</div>
</div>
<div class="footer">Lakehouse · Architecture page regenerates tests on every load · <a href="console">walkthrough</a> · <a href=".">dashboard</a></div>
<script>
var P=location.pathname.indexOf('/lakehouse')>=0?'/lakehouse':'';
var A=location.origin+P;
function el(tag, cls, text){
var e=document.createElement(tag);
if(cls) e.className=cls;
if(text!==undefined && text!==null) e.textContent=String(text);
return e;
}
function apiPost(path, body){
return fetch(A+path,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(body||{})}).then(function(r){return r.json()});
}
window.addEventListener('load',function(){
loadLiveSections();
});
function loadLiveSections(){
apiPost('/proof.json',{}).then(function(r){
var host1=document.getElementById('ch1-tests');host1.textContent='';
var tests=r.tests||[];
tests.forEach(function(t){
var row=el('div','row '+(t.pass?'pass':'fail'));
var left=document.createElement('div');left.style.flex='1';left.style.minWidth='0';
var title=el('div','title',(t.pass?'✓ ':'✗ ')+(t.name||'(unnamed)'));
var resultStr='';
if(typeof t.result==='object' && t.result) resultStr=JSON.stringify(t.result);
else if(t.result!==undefined && t.result!==null) resultStr=String(t.result);
var meta=el('div','meta',resultStr);
left.appendChild(title);left.appendChild(meta);
row.appendChild(left);
row.appendChild(el('div','val',(t.ms||0)+' ms'));
host1.appendChild(row);
});
var host6=document.getElementById('ch6-scale');host6.textContent='';
var scale=r.scale||{};
addStat(host6,(scale.total_rows||0).toLocaleString(),'Rows under management','across '+(scale.datasets||0)+' datasets','accent-b');
addStat(host6,(scale.total_chunks||0).toLocaleString(),'Vector chunks indexed','across '+(scale.indexes||0)+' HNSW + Lance indexes','accent-a');
var gpu=r.gpu||{};
var vramStr=(gpu.vram_used_mib!==undefined?gpu.vram_used_mib+' / '+gpu.vram_total_mib+' MiB':'—');
addStat(host6,vramStr,'GPU VRAM',(gpu.name||'A4000'),'accent-w');
if(r.recall){
var reco=r.recall||{};
var hnswStr=(reco.hnsw!==undefined?(reco.hnsw*100).toFixed(0)+'%':'—');
addStat(host6,hnswStr,'HNSW recall','measured · '+(reco.note||''),'accent-l');
}
var host6r=document.getElementById('ch6-recall');host6r.textContent='';
if(r.lance_10m){
var narr=el('div','narr');
narr.appendChild(el('strong',null,'Scale headroom: '));
narr.appendChild(document.createTextNode(
'Lance backend tested at '+r.lance_10m.vectors.toLocaleString()+' vectors, '
+r.lance_10m.disk_gb+' GB on disk, '+r.lance_10m.search_p50_ms+'ms p50. '
+(r.lance_10m.note||'')
));
host6r.appendChild(narr);
}
document.getElementById('hdr-time').textContent='Generated · '+(r.generated?new Date(r.generated).toLocaleTimeString():'live');
}).catch(function(e){
var host1=document.getElementById('ch1-tests');host1.textContent='';
host1.appendChild(el('div','err','Live tests failed: '+(e.message||e)+'. Server may be offline.'));
});
}
function addStat(host,n,l,sub,cls){
var d=el('div','card stat-lg '+(cls||''));
d.appendChild(el('div','n',n));
d.appendChild(el('div','l',l));
d.appendChild(el('div','sub',sub||''));
host.appendChild(d);
}
</script>
</body></html>
mcp-server/samples/staffing_roster_sample.csv
+26
View File
@@ -0,0 +1,26 @@
worker_id,name,role,city,state,email,phone,skills,certifications,availability,reliability,archetype
W-1001,Sarah Johnson,Forklift Operator,Chicago,IL,sarah.johnson@example.com,(312) 555-0101,pallet jack|hazmat|loading dock,OSHA-10|Forklift,0.92,0.88,reliable
W-1002,Michael Chen,Welder,Toledo,OH,m.chen@example.com,(419) 555-0234,TIG|MIG|blueprint reading|grinder,OSHA-10|AWS D1.1,0.71,0.94,specialist
W-1003,Maria Rodriguez,Warehouse Associate,Chicago,IL,maria.r@example.com,(312) 555-0311,inventory|RF scanner|pick-to-light,OSHA-10,0.85,0.80,reliable
W-1004,David Park,Machine Operator,Detroit,MI,dpark@example.com,(313) 555-0412,CNC|gauge R&R|lean manufacturing,OSHA-10|First Aid,0.60,0.91,specialist
W-1005,Jennifer Lopez,Loader,Chicago,IL,j.lopez@example.com,(312) 555-0501,loading dock|team lead|cold storage,OSHA-10|Forklift,0.95,0.76,communicator
W-1006,Robert Williams,Maintenance Tech,Milwaukee,WI,rwilliams@example.com,(414) 555-0612,electrical|PLC|hydraulics|CMMS,OSHA-10|LOTO,0.45,0.93,specialist
W-1007,Amanda Taylor,Quality Tech,Toledo,OH,a.taylor@example.com,(419) 555-0701,ISO 9001|calibration|root cause analysis,OSHA-10|Six Sigma Green,0.80,0.87,leader
W-1008,Carlos Mendoza,Electrician,Chicago,IL,c.mendoza@example.com,(312) 555-0808,conduit|motor controls|troubleshooting,OSHA-30|NEC,0.88,0.96,specialist
W-1009,Kim Nguyen,Forklift Operator,Detroit,MI,k.nguyen@example.com,(313) 555-0915,pallet jack|shipping|team lead,OSHA-10|Forklift,0.73,0.82,flexible
W-1010,James O'Brien,Welder,Milwaukee,WI,jobrien@example.com,(414) 555-1021,TIG|pipe welding|confined space,OSHA-10|AWS D1.1,0.90,0.79,reliable
W-1011,Priya Patel,Production Worker,Chicago,IL,priya.p@example.com,(312) 555-1118,line work|first article|labeling,OSHA-10,0.66,0.85,reliable
W-1012,Thomas Anderson,Material Handler,Toledo,OH,tanderson@example.com,(419) 555-1225,RF scanner|pallet jack|receiving,OSHA-10|Forklift,0.82,0.77,flexible
W-1013,Lisa Nakamura,Warehouse Associate,Detroit,MI,l.nakamura@example.com,(313) 555-1312,inventory|Excel|packaging,OSHA-10,0.77,0.89,communicator
W-1014,Brandon Moore,Machine Operator,Milwaukee,WI,b.moore@example.com,(414) 555-1405,CNC|SPC|conveyor ops,OSHA-10,0.56,0.94,specialist
W-1015,Emily Zhang,Assembler,Chicago,IL,emily.z@example.com,(312) 555-1502,assembly|gauge R&R|line lead,OSHA-10|Six Sigma Yellow,0.91,0.81,leader
W-1016,Marcus Johnson,Forklift Operator,Toledo,OH,mjohnson@example.com,(419) 555-1609,pallet jack|hazmat|overhead crane,OSHA-10|Forklift|Hazmat,0.68,0.88,reliable
W-1017,Anita Brooks,Line Lead,Detroit,MI,a.brooks@example.com,(313) 555-1715,team lead|training|SPC,OSHA-30|Six Sigma Green,0.52,0.97,leader
W-1018,Dmitri Volkov,Welder,Chicago,IL,d.volkov@example.com,(312) 555-1820,TIG|MIG|blueprint reading,OSHA-10|AWS D1.1|Confined Space,0.84,0.83,specialist
W-1019,Rachel Kim,Loader,Milwaukee,WI,r.kim@example.com,(414) 555-1911,loading dock|first aid|bilingual,OSHA-10|First Aid,0.79,0.84,communicator
W-1020,Samuel Park,Maintenance Tech,Toledo,OH,spark@example.com,(419) 555-2012,electrical|PLC|troubleshooting|CMMS,OSHA-10|LOTO,0.70,0.92,specialist
W-1021,Jordan Williams,Quality Tech,Detroit,MI,j.williams@example.com,(313) 555-2117,ISO 9001|calibration|SPC,OSHA-10|Six Sigma Green,0.83,0.86,leader
W-1022,Natalia Soto,Warehouse Associate,Chicago,IL,n.soto@example.com,(312) 555-2221,RF scanner|inventory|team lead,OSHA-10,0.89,0.90,communicator
W-1023,Henry Chen,Machine Operator,Milwaukee,WI,h.chen@example.com,(414) 555-2318,CNC|conveyor ops|root cause,OSHA-10,0.64,0.93,specialist
W-1024,Ava Martinez,Forklift Operator,Toledo,OH,a.martinez@example.com,(419) 555-2411,pallet jack|bilingual|cold storage,OSHA-10|Forklift,0.87,0.80,flexible
W-1025,Tyler Rodriguez,Production Worker,Chicago,IL,t.rodriguez@example.com,(312) 555-2515,line work|packaging|quality inspection,OSHA-10,0.74,0.86,reliable
1 worker_id name role city state email phone skills certifications availability reliability archetype
2 W-1001 Sarah Johnson Forklift Operator Chicago IL sarah.johnson@example.com (312) 555-0101 pallet jack|hazmat|loading dock OSHA-10|Forklift 0.92 0.88 reliable
3 W-1002 Michael Chen Welder Toledo OH m.chen@example.com (419) 555-0234 TIG|MIG|blueprint reading|grinder OSHA-10|AWS D1.1 0.71 0.94 specialist
4 W-1003 Maria Rodriguez Warehouse Associate Chicago IL maria.r@example.com (312) 555-0311 inventory|RF scanner|pick-to-light OSHA-10 0.85 0.80 reliable
5 W-1004 David Park Machine Operator Detroit MI dpark@example.com (313) 555-0412 CNC|gauge R&R|lean manufacturing OSHA-10|First Aid 0.60 0.91 specialist
6 W-1005 Jennifer Lopez Loader Chicago IL j.lopez@example.com (312) 555-0501 loading dock|team lead|cold storage OSHA-10|Forklift 0.95 0.76 communicator
7 W-1006 Robert Williams Maintenance Tech Milwaukee WI rwilliams@example.com (414) 555-0612 electrical|PLC|hydraulics|CMMS OSHA-10|LOTO 0.45 0.93 specialist
8 W-1007 Amanda Taylor Quality Tech Toledo OH a.taylor@example.com (419) 555-0701 ISO 9001|calibration|root cause analysis OSHA-10|Six Sigma Green 0.80 0.87 leader
9 W-1008 Carlos Mendoza Electrician Chicago IL c.mendoza@example.com (312) 555-0808 conduit|motor controls|troubleshooting OSHA-30|NEC 0.88 0.96 specialist
10 W-1009 Kim Nguyen Forklift Operator Detroit MI k.nguyen@example.com (313) 555-0915 pallet jack|shipping|team lead OSHA-10|Forklift 0.73 0.82 flexible
11 W-1010 James O'Brien Welder Milwaukee WI jobrien@example.com (414) 555-1021 TIG|pipe welding|confined space OSHA-10|AWS D1.1 0.90 0.79 reliable
12 W-1011 Priya Patel Production Worker Chicago IL priya.p@example.com (312) 555-1118 line work|first article|labeling OSHA-10 0.66 0.85 reliable
13 W-1012 Thomas Anderson Material Handler Toledo OH tanderson@example.com (419) 555-1225 RF scanner|pallet jack|receiving OSHA-10|Forklift 0.82 0.77 flexible
14 W-1013 Lisa Nakamura Warehouse Associate Detroit MI l.nakamura@example.com (313) 555-1312 inventory|Excel|packaging OSHA-10 0.77 0.89 communicator
15 W-1014 Brandon Moore Machine Operator Milwaukee WI b.moore@example.com (414) 555-1405 CNC|SPC|conveyor ops OSHA-10 0.56 0.94 specialist
16 W-1015 Emily Zhang Assembler Chicago IL emily.z@example.com (312) 555-1502 assembly|gauge R&R|line lead OSHA-10|Six Sigma Yellow 0.91 0.81 leader
17 W-1016 Marcus Johnson Forklift Operator Toledo OH mjohnson@example.com (419) 555-1609 pallet jack|hazmat|overhead crane OSHA-10|Forklift|Hazmat 0.68 0.88 reliable
18 W-1017 Anita Brooks Line Lead Detroit MI a.brooks@example.com (313) 555-1715 team lead|training|SPC OSHA-30|Six Sigma Green 0.52 0.97 leader
19 W-1018 Dmitri Volkov Welder Chicago IL d.volkov@example.com (312) 555-1820 TIG|MIG|blueprint reading OSHA-10|AWS D1.1|Confined Space 0.84 0.83 specialist
20 W-1019 Rachel Kim Loader Milwaukee WI r.kim@example.com (414) 555-1911 loading dock|first aid|bilingual OSHA-10|First Aid 0.79 0.84 communicator
21 W-1020 Samuel Park Maintenance Tech Toledo OH spark@example.com (419) 555-2012 electrical|PLC|troubleshooting|CMMS OSHA-10|LOTO 0.70 0.92 specialist
22 W-1021 Jordan Williams Quality Tech Detroit MI j.williams@example.com (313) 555-2117 ISO 9001|calibration|SPC OSHA-10|Six Sigma Green 0.83 0.86 leader
23 W-1022 Natalia Soto Warehouse Associate Chicago IL n.soto@example.com (312) 555-2221 RF scanner|inventory|team lead OSHA-10 0.89 0.90 communicator
24 W-1023 Henry Chen Machine Operator Milwaukee WI h.chen@example.com (414) 555-2318 CNC|conveyor ops|root cause OSHA-10 0.64 0.93 specialist
25 W-1024 Ava Martinez Forklift Operator Toledo OH a.martinez@example.com (419) 555-2411 pallet jack|bilingual|cold storage OSHA-10|Forklift 0.87 0.80 flexible
26 W-1025 Tyler Rodriguez Production Worker Chicago IL t.rodriguez@example.com (312) 555-2515 line work|packaging|quality inspection OSHA-10 0.74 0.86 reliable
mcp-server/search.html
+386 -16
View File
@@ -103,8 +103,12 @@ body{font-family:'Inter',-apple-system,system-ui,'Segoe UI',sans-serif;backgroun
<h1>Staffing Co-Pilot</h1> <h1>Staffing Co-Pilot</h1>
<nav> <nav>
<a href="." class="active">Dashboard</a> <a href="." class="active">Dashboard</a>
<a href="console">Intelligence Console</a> <a href="console">Walkthrough</a>
<a href="proof">Architecture</a> <a href="proof">Architecture</a>
<a href="spec">Spec</a>
<a href="onboard">Onboard</a>
<a href="alerts">Alerts</a>
<a href="workspaces">Workspaces</a>
</nav> </nav>
<div class="rt" id="status">Loading...</div> <div class="rt" id="status">Loading...</div>
</div> </div>
@@ -112,6 +116,22 @@ body{font-family:'Inter',-apple-system,system-ui,'Segoe UI',sans-serif;backgroun
<div id="main"><div class="ld">Analyzing contracts and workers...</div></div> <div id="main"><div class="ld">Analyzing contracts and workers...</div></div>
<div class="section" id="staffing-forecast-section">
<div class="section-header">
<span class="section-title">Staffing Forecast — Next 30 Days</span>
<span class="section-meta">Permits → predicted demand · Bench supply · Days to staffing deadline</span>
</div>
<div id="staffing-forecast"><div class="ld">Loading forecast...</div></div>
</div>
<div class="section" id="live-contracts-section">
<div class="section-header">
<span class="section-title">Live Contracts — Chicago Permits → Proposed Fills</span>
<span class="section-meta" id="live-contracts-meta">Real public permit data + worker bench + past playbook patterns</span>
</div>
<div id="live-contracts"><div class="ld">Loading live contracts...</div></div>
</div>
<div class="section" id="market-section"> <div class="section" id="market-section">
<div class="section-header"> <div class="section-header">
<span class="section-title">Market Intelligence</span> <span class="section-title">Market Intelligence</span>
@@ -131,7 +151,7 @@ body{font-family:'Inter',-apple-system,system-ui,'Segoe UI',sans-serif;backgroun
<div class="section"> <div class="section">
<div class="section-header"> <div class="section-header">
<span class="section-title">Worker Search</span> <span class="section-title">Worker Search</span>
<span class="section-meta">Natural language · 500K profiles</span> <span class="section-meta" id="worker-search-meta">Natural language search</span>
</div> </div>
<details class="sa" open><summary>Search all workers</summary><div class="inner"> <details class="sa" open><summary>Search all workers</summary><div class="inner">
<input type="text" id="sq" placeholder="Try: reliable forklift operator available in Nashville" onkeydown="if(event.key==='Enter')doSearch()"> <input type="text" id="sq" placeholder="Try: reliable forklift operator available in Nashville" onkeydown="if(event.key==='Enter')doSearch()">
@@ -140,19 +160,209 @@ body{font-family:'Inter',-apple-system,system-ui,'Segoe UI',sans-serif;backgroun
<button class="sbtn" onclick="doSearch()">Find Workers</button><div id="sresults"></div></div></details> <button class="sbtn" onclick="doSearch()">Find Workers</button><div id="sresults"></div></div></details>
</div> </div>
<div class="ft">Staffing Co-Pilot · Hybrid SQL + Vector Search · 500K embedded profiles · <a href="console">Console</a> · <a href="proof">Architecture</a></div> <div class="ft" id="footer">Staffing Co-Pilot · Hybrid SQL + Vector Search · loading scale… · <a href="console">Console</a> · <a href="proof">Architecture</a></div>
</div> </div>
<script> <script>
var P=location.pathname.indexOf('/lakehouse')>=0?'/lakehouse':''; var P=location.pathname.indexOf('/lakehouse')>=0?'/lakehouse':'';
var A=location.origin+P; var A=location.origin+P;
var AC=['#1a2744','#1a3a2a','#2a1a3a','#3a2a1a','#1a3a3a','#2a2a1a']; var AC=['#1a2744','#1a3a2a','#2a1a3a','#3a2a1a','#1a3a3a','#2a2a1a'];
var lastQuery=''; var lastQuery='';
window.addEventListener('load',function(){loadDay();loadMarket();loadLearning()}); window.addEventListener('load',function(){loadSystemSummary();loadDay();loadStaffingForecast();loadLiveContracts();loadMarket();loadLearning()});
function loadSystemSummary(){
api('/system/summary',{}).then(function(s){
if(!s) return;
var totalRows=(s.total_rows||0).toLocaleString();
var workers=(s.workers_500k_rows||0).toLocaleString();
var chunks=(s.total_chunks||0).toLocaleString();
var ds=s.datasets||0;
var meta1=document.getElementById('live-contracts-meta');
if(meta1) meta1.textContent='Real public permit data + '+workers+' worker bench + past playbook patterns';
var meta2=document.getElementById('worker-search-meta');
if(meta2) meta2.textContent='Natural language · '+workers+' profiles across '+ds+' datasets';
var foot=document.getElementById('footer');
if(foot){
foot.textContent='';
foot.appendChild(document.createTextNode('Staffing Co-Pilot · Hybrid SQL + Vector Search · '+totalRows+' rows across '+ds+' datasets · '+chunks+' vector chunks · '));
var a1=document.createElement('a');a1.href='console';a1.textContent='Console';foot.appendChild(a1);
foot.appendChild(document.createTextNode(' · '));
var a2=document.createElement('a');a2.href='proof';a2.textContent='Architecture';foot.appendChild(a2);
}
// Also update the collapsible search box label if not yet populated
var sum=document.querySelector('.sa summary');
if(sum&&/Search all\s*\d*\s*workers/.test(sum.textContent)){
sum.textContent='Search all '+workers+' workers';
}
}).catch(function(){/* non-fatal */});
}
function loadStaffingForecast(){
api('/intelligence/staffing_forecast',{}).then(function(r){
var el=document.getElementById('staffing-forecast');el.textContent='';
if(!r||!r.forecast){el.textContent='Forecast unavailable.';return}
// Header summary
var hdr=document.createElement('div');hdr.style.cssText='background:#0d1117;border:1px solid #171d27;border-radius:10px;padding:14px;margin-bottom:10px;display:flex;justify-content:space-between;gap:16px';
var left=document.createElement('div');
var big=document.createElement('div');big.style.cssText='font-size:22px;font-weight:700;color:#e6edf3;letter-spacing:-0.5px';
big.textContent='$'+(r.total_cost||0).toLocaleString('en-US',{maximumFractionDigits:0})+' in construction coming';
var sub=document.createElement('div');sub.style.cssText='color:#8b949e;font-size:12px;margin-top:4px';
sub.textContent=r.permit_count+' permits filed last 30 days · ~'+r.total_estimated_workers+' workers needed across roles';
left.appendChild(big);left.appendChild(sub);hdr.appendChild(left);
var right=document.createElement('div');right.style.cssText='text-align:right;font-size:11px';
if(r.critical_roles>0){
var c=document.createElement('div');c.style.cssText='color:#f85149;font-weight:700';
c.textContent=r.critical_roles+' CRITICAL role'+(r.critical_roles!==1?'s':'')+' — supply gap';right.appendChild(c);
} else if(r.tight_roles>0){
var t=document.createElement('div');t.style.cssText='color:#d29922;font-weight:700';
t.textContent=r.tight_roles+' tight role'+(r.tight_roles!==1?'s':'');right.appendChild(t);
} else {
var g=document.createElement('div');g.style.cssText='color:#3fb950;font-weight:600';g.textContent='bench covers predicted demand';right.appendChild(g);
}
var when=document.createElement('div');when.style.cssText='color:#545d68;margin-top:2px';
when.textContent='updated '+((r.duration_ms||0)/1000).toFixed(1)+'s ago';right.appendChild(when);
hdr.appendChild(right);el.appendChild(hdr);
// Per-role rows
r.forecast.forEach(function(f){
var row=document.createElement('div');
var riskColor={critical:'#f85149',tight:'#d29922',watch:'#d29922',ok:'#3fb950'}[f.risk]||'#8b949e';
row.style.cssText='background:#0d1117;border:1px solid #171d27;border-radius:8px;padding:10px 14px;margin-bottom:6px;border-left:3px solid '+riskColor+';display:flex;justify-content:space-between;gap:12px;font-size:12px;align-items:center';
var l=document.createElement('div');
var n=document.createElement('div');n.style.cssText='color:#e6edf3;font-weight:600;font-size:13px';
n.textContent=f.role;
var d=document.createElement('div');d.style.cssText='color:#8b949e;font-size:11px;margin-top:2px';
d.textContent=f.demand_permits+' permit'+(f.demand_permits!==1?'s':'')+' · est '+f.demand_workers+' workers · earliest staffing deadline '+f.earliest_staffing_deadline;
l.appendChild(n);l.appendChild(d);row.appendChild(l);
var r2=document.createElement('div');r2.style.cssText='text-align:right;white-space:nowrap';
var cov=document.createElement('div');cov.style.cssText='color:'+riskColor+';font-weight:700;font-size:13px';
cov.textContent=f.bench_available.toLocaleString()+' / '+f.demand_workers+' available ('+f.coverage_pct+'%)';
var days=document.createElement('div');days.style.cssText='color:'+(f.days_to_deadline<=0?'#f85149':f.days_to_deadline<=7?'#d29922':'#8b949e')+';font-size:11px;margin-top:2px';
days.textContent=f.days_to_deadline<=0?(Math.abs(f.days_to_deadline)+'d overdue'):(f.days_to_deadline+'d to deadline');
r2.appendChild(cov);r2.appendChild(days);row.appendChild(r2);
el.appendChild(row);
});
}).catch(function(e){
document.getElementById('staffing-forecast').textContent='Forecast error: '+(e.message||e);
});
}
function api(path,body){ function api(path,body){
return fetch(A+path,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(body)}).then(function(r){return r.json()}) return fetch(A+path,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(body)}).then(function(r){return r.json()})
} }
// Mirror of the Bun-side pay-rate formula so client-side renderers
// (main search, modal) can show a rate when they only have worker
// fields, not a pre-enriched hybrid source. Keep in sync with
// impliedPayRate() in mcp-server/index.ts.
var ROLE_BASE_PAY={Electrician:28,Welder:26,'Machine Operator':24,'Maintenance Tech':26,
'Forklift Operator':20,Loader:17,'Warehouse Associate':17,'Material Handler':18,
'Production Worker':18,'Quality Tech':23,'Line Lead':22,Assembler:18,'Shipping Clerk':19};
function computeImpliedPayRate(role,rel,archetype){
var base=ROLE_BASE_PAY[role||'']||19;
var r=typeof rel==='string'?parseFloat(rel):(rel||0.5);
var relBump=(isFinite(r)?r:0.5)*4;
var a=(archetype||'').toLowerCase();
var archBump=a==='specialist'?4:a==='leader'?3:a==='reliable'?1:0;
return Math.round((base+relBump+archBump)*100)/100;
}
function loadLiveContracts(){
// Pair live Chicago permits with our 500K worker bench and the
// meta-index discovered patterns for each role+geo. This is the
// "real external data meets synthetic playbook learning" card set.
api('/intelligence/permit_contracts',{}).then(function(r){
var el=document.getElementById('live-contracts');el.textContent='';
if(!r||!r.contracts||r.contracts.length===0){
el.textContent='No permits returned.';return;
}
r.contracts.forEach(function(c){
var p=c.permit||{}, prop=c.proposed||{}, tl=c.timeline||{};
var urg=tl.urgency||'scheduled';
var borderColor={overdue:'#f85149',urgent:'#d29922',soon:'#388bfd',scheduled:'#2ea043'}[urg]||'#388bfd';
var card=document.createElement('div');card.className='insight info';
card.style.cssText='background:#0d1117;border:1px solid #171d27;border-radius:10px;padding:16px;margin-bottom:10px;border-left:3px solid '+borderColor;
// Header — permit
var hdr=document.createElement('div');hdr.style.cssText='display:flex;justify-content:space-between;margin-bottom:8px;gap:12px';
var left=document.createElement('div');
var title=document.createElement('div');title.style.cssText='font-weight:600;color:#e6edf3;font-size:14px';
title.textContent='$'+(p.cost||0).toLocaleString()+' · '+(p.work_type||'');
var addr=document.createElement('div');addr.style.cssText='color:#8b949e;font-size:12px;margin-top:2px';
addr.textContent=(p.address||'')+' · Chicago, IL · filed '+(p.issue_date||'');
// Timeline chip
if(tl.days_to_deadline!==undefined){
var tmline=document.createElement('div');tmline.style.cssText='color:'+borderColor+';font-size:11px;font-weight:600;margin-top:4px';
var urgLabel={overdue:'OVERDUE',urgent:'URGENT',soon:'SOON',scheduled:'SCHEDULED'}[urg]||'SCHEDULED';
var dd=tl.days_to_deadline;
var txt=urgLabel+' · staffing window opens '+(tl.staffing_window_opens||'')+' ('+(dd<=0?Math.abs(dd)+'d overdue':dd+'d to deadline')+') · construction est '+(tl.estimated_construction_start||'');
tmline.textContent=txt;addr.appendChild(document.createElement('br'));left.appendChild(tmline);
}
left.appendChild(title);left.appendChild(addr);
var right=document.createElement('div');right.style.cssText='color:#58a6ff;font-size:12px;font-weight:600;text-align:right;white-space:nowrap';
right.textContent=prop.count+'× '+prop.role;
var sub=document.createElement('div');sub.style.cssText='color:#545d68;font-size:10px;text-align:right';
sub.textContent='pool: '+(prop.pool_size||'?').toLocaleString()+' available';
right.appendChild(sub);
// Rate awareness: show implied bill rate per contract
if(c.implied_bill_rate){
var rate=document.createElement('div');
rate.style.cssText='color:#d29922;font-size:10px;text-align:right;margin-top:3px';
rate.textContent='bill rate: $'+c.implied_bill_rate.toFixed(2)+'/hr';
right.appendChild(rate);
}
hdr.appendChild(left);hdr.appendChild(right);card.appendChild(hdr);
// Description
if(p.description){
var desc=document.createElement('div');desc.style.cssText='color:#94a3b8;font-size:11px;margin-bottom:10px;line-height:1.5';
desc.textContent=p.description;card.appendChild(desc);
}
// Pattern (meta-index) chip
if(c.discovered_pattern){
var pat=document.createElement('div');pat.style.cssText='background:#0d2818;border:1px solid #2ea04360;border-radius:6px;padding:8px 12px;margin-bottom:10px;font-size:11px;color:#86efac;line-height:1.5';
var plabel=document.createElement('span');plabel.style.cssText='color:#3fb950;font-weight:600;margin-right:6px';
plabel.textContent='MEMORY ('+c.pattern_matched+' playbooks):';
pat.appendChild(plabel);
pat.appendChild(document.createTextNode(' '+c.discovered_pattern));
card.appendChild(pat);
}
// Candidates
var cands=prop.candidates||[];
cands.slice(0,3).forEach(function(cand,i){
var row=document.createElement('div');row.style.cssText='display:flex;align-items:center;gap:10px;padding:6px 10px;background:#161b22;border-radius:6px;margin-bottom:4px;font-size:12px';
var av=document.createElement('div');av.style.cssText='width:28px;height:28px;border-radius:6px;display:flex;align-items:center;justify-content:center;font-weight:600;font-size:10px;color:#e6edf3;background:'+AC[i%AC.length];
av.textContent=(cand.name||'?').split(' ').map(function(n){return (n[0]||'').toUpperCase()}).join('').substring(0,2);
var info=document.createElement('div');info.style.cssText='flex:1;min-width:0';
var nm=document.createElement('div');nm.style.cssText='color:#e6edf3;font-weight:500';nm.textContent=cand.name||cand.doc_id;
if((cand.playbook_boost||0)>0){
var chip=document.createElement('span');chip.style.cssText='margin-left:8px;padding:2px 7px;border-radius:9px;font-size:9px;font-weight:600;background:#0d2818;border:1px solid #2ea043;color:#3fb950;vertical-align:middle';
chip.textContent='Endorsed · '+(cand.playbook_citations||[]).length+' playbook'+((cand.playbook_citations||[]).length===1?'':'s');
nm.appendChild(chip);
}
// Rate warning chip when worker's pay exceeds the contract's bill rate
if(cand.over_bill_rate){
var warn=document.createElement('span');warn.style.cssText='margin-left:6px;padding:2px 7px;border-radius:9px;font-size:9px;font-weight:600;background:#3a1a1a;border:1px solid #f85149;color:#fca5a5;vertical-align:middle';
warn.textContent='Over bill rate';
warn.title='Worker\'s implied pay rate ($'+(cand.implied_pay_rate||0).toFixed(2)+'/hr) exceeds contract bill rate ($'+(c.implied_bill_rate||0).toFixed(2)+'/hr) — margin at risk';
nm.appendChild(warn);
}
var sub2=document.createElement('div');sub2.style.cssText='color:#545d68;font-size:10px';
var subText=cand.doc_id+' · score '+(cand.score||0).toFixed(3);
if(cand.implied_pay_rate) subText+=' · pay $'+cand.implied_pay_rate.toFixed(2)+'/hr';
sub2.textContent=subText;
info.appendChild(nm);info.appendChild(sub2);
row.appendChild(av);row.appendChild(info);
card.appendChild(row);
});
if(cands.length>3){
var more=document.createElement('div');more.style.cssText='font-size:10px;color:#58a6ff;padding:4px 10px;margin-top:2px';
more.textContent='+ '+(cands.length-3)+' more candidates available';
card.appendChild(more);
}
el.appendChild(card);
});
}).catch(function(e){
document.getElementById('live-contracts').textContent='Error loading: '+e.message;
});
}
function loadDay(){ function loadDay(){
// Step 1: run simulation + get real worker count + populate dropdowns from actual data // Step 1: run simulation + get real worker count + populate dropdowns from actual data
Promise.all([ Promise.all([
@@ -539,11 +749,84 @@ function showProfile(workerData){
body.appendChild(srcBox); body.appendChild(srcBox);
} }
// Call history — recruiter-facing institutional memory from call_log.
// Queries for prior contact with this specific worker (by name
// cross-ref). Fails soft: if no rows, shows "no recent contact" which
// is itself a useful signal (or an honest tell about data sparsity).
addSection(body,'Recent Contact','Last phone outreach logged in call_log');
var callBox=document.createElement('div');
callBox.style.cssText='background:#0d1117;border-radius:8px;padding:14px;margin-bottom:20px;font-size:12px;color:#8b949e;line-height:1.6';
callBox.textContent='Checking call log...';body.appendChild(callBox);
var nameLitC=(workerData.nm||'').replace(/'/g,"''");
var callSQL="SELECT cl.timestamp, cl.recruiter, cl.duration_seconds, cl.disposition "
+"FROM call_log cl JOIN candidates c ON c.candidate_id = cl.candidate_id "
+"WHERE CONCAT(c.first_name, ' ', c.last_name) = '"+nameLitC+"' "
+"ORDER BY cl.timestamp DESC LIMIT 3";
api('/sql',{sql:callSQL}).then(function(r){
callBox.textContent='';
var rows=(r&&r.rows)||[];
if(rows.length===0){
callBox.textContent='No recent call logged for '+(workerData.nm||'this worker')+'. Data note: call_log cross-references candidate IDs that may not align with workers_500k — real ATS integration required for full coverage.';
callBox.style.color='#484f58';return;
}
rows.forEach(function(c){
var row=document.createElement('div');row.style.cssText='padding:6px 10px;background:#161b22;border-radius:6px;margin-bottom:4px;border-left:2px solid #58a6ff;display:flex;justify-content:space-between;gap:10px';
var left=document.createElement('div');
var ts=(c.timestamp||'').substring(0,10);
var dur=Math.round((c.duration_seconds||0)/60);
var l1=document.createElement('div');l1.style.cssText='color:#e6edf3;font-weight:500;font-size:12px';
l1.textContent=ts+(c.recruiter?' · by '+c.recruiter:'');left.appendChild(l1);
var l2=document.createElement('div');l2.style.cssText='color:#8b949e;font-size:10px';
l2.textContent=(c.disposition||'?').replace(/_/g,' ')+(dur?' · '+dur+' min':'');left.appendChild(l2);
row.appendChild(left);callBox.appendChild(row);
});
}).catch(function(){callBox.textContent='(call log unavailable)';callBox.style.color='#484f58'});
// Past playbook history — Phase 19 institutional memory surfaced on
// the worker's own profile. Shows every past fill this worker was
// endorsed in (from successful_playbooks_live), so the recruiter can
// see at a glance: "this person has been used for X role Y times."
addSection(body,'Past Playbooks','Where this worker has been endorsed before');
var histBox=document.createElement('div');histBox.id='hist-'+(workerData.nm||'anon').replace(/\s/g,'-');
histBox.style.cssText='background:#0d1117;border-radius:8px;padding:14px;margin-bottom:20px;font-size:12px;color:#8b949e;line-height:1.6';
histBox.textContent='Loading history...';body.appendChild(histBox);
var city=(workerData.loc||'').split(',')[0].trim();
var state=(workerData.loc||'').split(',').pop().trim();
var nameLit=(workerData.nm||'').replace(/'/g,"''");
var sqlQ="SELECT operation, approach, context, timestamp FROM successful_playbooks_live "
+"WHERE result LIKE '%"+nameLit+"%' ORDER BY timestamp DESC LIMIT 8";
api('/sql',{sql:sqlQ}).then(function(r){
histBox.textContent='';
var rows=(r&&r.rows)||[];
if(rows.length===0){
histBox.textContent='No prior playbooks for '+(workerData.nm||'this worker')+' yet. First placement builds the first entry.';
histBox.style.color='#484f58';return;
}
var hdr2=document.createElement('div');hdr2.style.cssText='color:#3fb950;font-weight:600;margin-bottom:8px;font-size:11px';
hdr2.textContent=rows.length+' past endorsement'+(rows.length!==1?'s':'');
histBox.appendChild(hdr2);
rows.forEach(function(pb){
var row=document.createElement('div');row.style.cssText='padding:6px 10px;background:#161b22;border-radius:6px;margin-bottom:4px;border-left:2px solid #2ea043';
var op=document.createElement('div');op.style.cssText='color:#e6edf3;font-weight:500;font-size:12px';
op.textContent=pb.operation||'(unknown op)';row.appendChild(op);
var meta=document.createElement('div');meta.style.cssText='color:#8b949e;font-size:10px;margin-top:2px';
var ts=(pb.timestamp||'').substring(0,10);
meta.textContent=ts+' · '+(pb.approach||'').slice(0,40)+(pb.context?' · '+pb.context.slice(0,30):'');
row.appendChild(meta);histBox.appendChild(row);
});
}).catch(function(){
histBox.textContent='(history unavailable)';histBox.style.color='#484f58';
});
// Actions // Actions
var acts=document.createElement('div');acts.style.cssText='display:flex;gap:8px;padding-top:16px;border-top:1px solid #21262d'; var acts=document.createElement('div');acts.style.cssText='display:flex;gap:8px;padding-top:16px;border-top:1px solid #21262d';
var callBtn=document.createElement('button');callBtn.style.cssText='flex:1;padding:12px;border-radius:8px;font-size:14px;font-weight:600;cursor:pointer;border:none;background:#1f3d68;color:#58a6ff';callBtn.textContent='Call'; var callBtn=document.createElement('button');callBtn.style.cssText='flex:1;padding:12px;border-radius:8px;font-size:14px;font-weight:600;cursor:pointer;border:none;background:#1f3d68;color:#58a6ff';callBtn.textContent='Call';
callBtn.onclick=function(){logAction(workerData,'call',callBtn)};
var smsBtn=document.createElement('button');smsBtn.style.cssText='flex:1;padding:12px;border-radius:8px;font-size:14px;font-weight:600;cursor:pointer;border:none;background:#0d261a;color:#3fb950';smsBtn.textContent='Send SMS'; var smsBtn=document.createElement('button');smsBtn.style.cssText='flex:1;padding:12px;border-radius:8px;font-size:14px;font-weight:600;cursor:pointer;border:none;background:#0d261a;color:#3fb950';smsBtn.textContent='Send SMS';
acts.appendChild(callBtn);acts.appendChild(smsBtn);body.appendChild(acts); smsBtn.onclick=function(){logAction(workerData,'sms',smsBtn)};
var noshowBtn=document.createElement('button');noshowBtn.style.cssText='flex:1;padding:12px;border-radius:8px;font-size:14px;font-weight:600;cursor:pointer;border:none;background:#3a1a1a;color:#f85149';noshowBtn.textContent='No-show';
noshowBtn.onclick=function(){logAction(workerData,'failure',noshowBtn)};
acts.appendChild(callBtn);acts.appendChild(smsBtn);acts.appendChild(noshowBtn);body.appendChild(acts);
modal.appendChild(body);overlay.appendChild(modal);document.body.appendChild(overlay); modal.appendChild(body);overlay.appendChild(modal);document.body.appendChild(overlay);
} }
@@ -580,14 +863,35 @@ function addWorkerInsight(parent,name,detail,why,idx,highlight){
var info=document.createElement('div');info.className='info'; var info=document.createElement('div');info.className='info';
var nm=document.createElement('div');nm.className='nm';nm.textContent=name; var nm=document.createElement('div');nm.className='nm';nm.textContent=name;
// Phase 19: when a past playbook endorsed this worker, show a green chip // Phase 19: when a past playbook endorsed this worker, show a green chip
// next to the name. Hover reveals the citation IDs. // next to the name. Hover reveals a NARRATIVE of past endorsements
// derived from successful_playbooks_live — "filled X in Y on date" —
// rather than opaque pb-seed-xxx ids. Recruiters need stories, not
// citation keys. Lazy-loaded per card on first render.
if(boostInfo && boostInfo.boost > 0){ if(boostInfo && boostInfo.boost > 0){
var chip=document.createElement('span'); var chip=document.createElement('span');
chip.style.cssText='display:inline-block;margin-left:8px;padding:2px 7px;border-radius:9px;font-size:10px;font-weight:600;background:#0d2818;border:1px solid #2ea043;color:#3fb950;vertical-align:middle'; chip.style.cssText='display:inline-block;margin-left:8px;padding:2px 7px;border-radius:9px;font-size:10px;font-weight:600;background:#0d2818;border:1px solid #2ea043;color:#3fb950;vertical-align:middle;cursor:help';
var n=(boostInfo.citations && boostInfo.citations.length) || 0; var n=(boostInfo.citations && boostInfo.citations.length) || 0;
chip.textContent='Endorsed · '+n+' playbook'+(n!==1?'s':''); chip.textContent='Endorsed · '+n+' playbook'+(n!==1?'s':'');
chip.title='Boosted by past playbooks: '+(boostInfo.citations||[]).join(', '); chip.title='Loading past playbooks for '+name+'...';
nm.appendChild(chip); nm.appendChild(chip);
// Fetch narrative for this worker lazily
var safeName = (name||'').replace(/'/g,"''");
var narrativeSQL = "SELECT operation, result, timestamp FROM successful_playbooks_live "
+ "WHERE result LIKE '%"+safeName+"%' ORDER BY timestamp DESC LIMIT 5";
api('/sql',{sql:narrativeSQL}).then(function(r){
var rows=(r&&r.rows)||[];
if(rows.length===0){
chip.title=name+' — endorsed in '+n+' playbook'+(n!==1?'s':'')+' (narrative unavailable — may have been seeded without SQL persistence)';
return;
}
var stories=rows.map(function(pb){
var d=(pb.timestamp||'').substring(0,10);
return '• '+(pb.operation||'?').replace(/^fill:\s*/,'')+' ('+d+')';
});
chip.title=name+' — past endorsements:\n'+stories.join('\n');
}).catch(function(){
chip.title=name+' — endorsed in '+n+' playbook'+(n!==1?'s':'');
});
} }
var dt=document.createElement('div');dt.className='detail';dt.textContent=detail; var dt=document.createElement('div');dt.className='detail';dt.textContent=detail;
info.appendChild(nm);info.appendChild(dt); info.appendChild(nm);info.appendChild(dt);
@@ -595,10 +899,16 @@ function addWorkerInsight(parent,name,detail,why,idx,highlight){
w.appendChild(info); w.appendChild(info);
var acts=document.createElement('div');acts.className='acts'; var acts=document.createElement('div');acts.className='acts';
var call=document.createElement('button');call.className='ibtn call';call.textContent='Call'; var call=document.createElement('button');call.className='ibtn call';call.textContent='Call';
call.onclick=function(e){e.stopPropagation();logSelection(workerDataRef)}; call.onclick=function(e){e.stopPropagation();logAction(workerDataRef,'call',call)};
var sms=document.createElement('button');sms.className='ibtn sms';sms.textContent='SMS'; var sms=document.createElement('button');sms.className='ibtn sms';sms.textContent='SMS';
sms.onclick=function(e){e.stopPropagation();logSelection(workerDataRef)}; sms.onclick=function(e){e.stopPropagation();logAction(workerDataRef,'sms',sms)};
acts.appendChild(call);acts.appendChild(sms);w.appendChild(acts); // Negative-signal button — recruiter marks a worker as "didn't work out"
// which fires /log_failure. Each such mark dampens that worker's
// future boost in the same geo by 0.5^n.
var noshow=document.createElement('button');noshow.className='ibtn';noshow.textContent='No-show';
noshow.style.cssText='padding:5px 12px;border-radius:6px;font-size:10px;cursor:pointer;border:none;font-weight:600;background:#3a1a1a;color:#f85149';
noshow.onclick=function(e){e.stopPropagation();logAction(workerDataRef,'failure',noshow)};
acts.appendChild(call);acts.appendChild(sms);acts.appendChild(noshow);w.appendChild(acts);
parent.appendChild(w); parent.appendChild(w);
} }
@@ -644,6 +954,32 @@ function doSearch(){
var h=document.createElement('div');h.style.cssText='color:#8b949e;font-size:12px;margin-bottom:10px'; var h=document.createElement('div');h.style.cssText='color:#8b949e;font-size:12px;margin-bottom:10px';
h.textContent=(d.sql_matches?d.sql_matches.toLocaleString()+' workers matched — ':'')+'showing best results ('+(d.duration_ms||0)+'ms)'; h.textContent=(d.sql_matches?d.sql_matches.toLocaleString()+' workers matched — ':'')+'showing best results ('+(d.duration_ms||0)+'ms)';
out.appendChild(h); out.appendChild(h);
// Meta-index signal — ALWAYS render when the system has any memory,
// even if no trait crossed threshold. Silence here would have
// recruiters assume "no signal" when the reality is "threshold
// filtered it out" or "memory is sparse for this geo." Trust
// depends on the system being honest about what it doesn't know.
if(d.pattern_playbooks_matched > 0 || d.discovered_pattern){
var mem=document.createElement('div');
mem.style.cssText='background:#0d2818;border:1px solid #2ea04360;border-radius:6px;padding:8px 12px;margin-bottom:10px;font-size:11px;color:#86efac;line-height:1.5';
var label=document.createElement('span');label.style.cssText='color:#3fb950;font-weight:600;margin-right:6px';
label.textContent='MEMORY ('+(d.pattern_playbooks_matched||0)+' playbook'+(d.pattern_playbooks_matched===1?'':'s')+'):';
mem.appendChild(label);
var pattern = d.discovered_pattern || '';
if(!pattern || pattern.indexOf('No similar')>=0 || pattern.indexOf('0 workers')>=0){
mem.appendChild(document.createTextNode(' memory is sparse for this role+geo — no trait crossed threshold. Will accumulate as fills land.'));
mem.style.color='#6ca885';
} else {
mem.appendChild(document.createTextNode(' '+pattern));
}
out.appendChild(mem);
} else {
// Zero playbooks matched — be explicit
var mem0=document.createElement('div');
mem0.style.cssText='background:#161b22;border:1px solid #21262d;border-radius:6px;padding:6px 12px;margin-bottom:10px;font-size:11px;color:#6e7681';
mem0.textContent='MEMORY: no similar past playbooks yet — first fill of this kind will seed it.';
out.appendChild(mem0);
}
// Render results based on type // Render results based on type
var workers=d.sql_results||[]; var workers=d.sql_results||[];
if(workers.length){ if(workers.length){
@@ -656,6 +992,10 @@ function doSearch(){
var why='Reliability: '+Math.round((w.rel||0)*100)+'%'; var why='Reliability: '+Math.round((w.rel||0)*100)+'%';
if(w.avail)why+=' · Available: '+Math.round(w.avail*100)+'%'; if(w.avail)why+=' · Available: '+Math.round(w.avail*100)+'%';
if(w.archetype)why+=' · '+w.archetype; if(w.archetype)why+=' · '+w.archetype;
// Derive and show implied pay rate client-side so the main search
// surface matches the live-contracts cards. Same formula as Bun.
var rate=computeImpliedPayRate(w.role,w.rel,w.archetype);
if(rate) why+=' · pay $'+rate.toFixed(2)+'/hr';
addWorkerInsight(out,w.name,detail.join(' · '),why,i,null,wd); addWorkerInsight(out,w.name,detail.join(' · '),why,i,null,wd);
}); });
} else { } else {
@@ -781,12 +1121,42 @@ function loadMarket(){
} }
// ─── Learning Loop ─── // ─── Learning Loop ───
function logSelection(workerData){ // Real recruiter actions feed the Phase 19 feedback chain directly:
if(!lastQuery||!workerData)return; // Call/SMS → /log → /vectors/playbook_memory/seed (positive endorsement)
fetch(A+'/intelligence/learn',{method:'POST',headers:{'Content-Type':'application/json'}, // No-show → /log_failure → /vectors/playbook_memory/mark_failed (penalty)
body:JSON.stringify({query:lastQuery,worker_name:workerData.nm,worker_id:workerData.nm,worker_role:workerData.role,worker_state:(workerData.loc||'').split(',').pop().trim(),worker_city:(workerData.loc||'').split(',')[0].trim(),filters:document.getElementById('sst').value+' '+document.getElementById('srl').value}) // Every click trains the system; the next search boosts/dampens accordingly.
}).then(function(){loadLearning()}).catch(function(){}); function logAction(workerData, kind, btnEl){
if(!workerData)return;
var role=workerData.role||'Worker';
var city=(workerData.loc||'').split(',')[0].trim();
var state=(workerData.loc||'').split(',').pop().trim();
if(!city||!state){flashBtn(btnEl,'no geo');return;}
var op='fill: '+role+' x1 in '+city+', '+state;
if(kind==='failure'){
fetch(A+'/log_failure',{method:'POST',headers:{'Content-Type':'application/json'},
body:JSON.stringify({operation:op,failed_names:[workerData.nm],reason:'marked no-show via UI'})
}).then(function(r){return r.json()}).then(function(d){
flashBtn(btnEl, d&&d.marked?'Flagged':'Ghost');
loadLearning();
}).catch(function(){flashBtn(btnEl,'err')});
} else {
fetch(A+'/log',{method:'POST',headers:{'Content-Type':'application/json'},
body:JSON.stringify({operation:op,approach:kind+' from UI',
result:'1/1 filled → '+workerData.nm,
context:'client=ui query='+(lastQuery||'(direct)').slice(0,40)})
}).then(function(r){return r.json()}).then(function(d){
flashBtn(btnEl, d&&d.seeded?'Logged':'Ghost');
loadLearning();
}).catch(function(){flashBtn(btnEl,'err')});
}
} }
function flashBtn(btn,label){
if(!btn)return;
var old=btn.textContent;btn.textContent=label;btn.disabled=true;
setTimeout(function(){btn.textContent=old;btn.disabled=false},1400);
}
// Back-compat shim — any legacy caller still pointing at logSelection.
function logSelection(workerData){ logAction(workerData, 'call', null); }
function loadLearning(){ function loadLearning(){
api('/intelligence/activity',{}).then(function(d){ api('/intelligence/activity',{}).then(function(d){
mcp-server/spec.html
+501
View File
@@ -0,0 +1,501 @@
<!DOCTYPE html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>Lakehouse — Technical Specification</title>
<style>
*{margin:0;padding:0;box-sizing:border-box}
body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;color:#b0b8c4;font-size:14px;line-height:1.6;-webkit-font-smoothing:antialiased}
a{color:#58a6ff;text-decoration:none}
a:hover{color:#79c0ff}
.bar{background:#0d1117;padding:0 24px;height:56px;border-bottom:1px solid #171d27;display:flex;justify-content:space-between;align-items:center;position:sticky;top:0;z-index:10}
.bar h1{font-size:14px;font-weight:600;color:#e6edf3;letter-spacing:-0.2px}
.bar nav{display:flex;gap:2px}
.bar nav a{font-size:12px;color:#545d68;padding:6px 14px;border-radius:6px;transition:all 0.15s}
.bar nav a:hover{color:#e6edf3;background:#161b22}
.bar nav a.active{color:#e6edf3;background:#1c2333}
.bar .rt{font-size:11px;color:#545d68}
.layout{display:grid;grid-template-columns:220px 1fr;gap:0;max-width:1200px;margin:0 auto}
.toc{position:sticky;top:72px;align-self:start;padding:24px 12px;border-right:1px solid #171d27;height:calc(100vh - 72px);overflow-y:auto}
.toc .hdr{color:#545d68;font-size:10px;text-transform:uppercase;letter-spacing:1.4px;font-weight:600;margin-bottom:10px;padding-left:8px}
.toc a{display:block;color:#8b949e;font-size:12px;padding:6px 10px;border-radius:6px;margin-bottom:2px;line-height:1.4}
.toc a:hover{color:#e6edf3;background:#161b22}
.wrap{padding:28px 24px 60px;min-width:0}
.chapter{margin-bottom:48px;scroll-margin-top:72px}
.chapter .num{color:#545d68;font-size:11px;font-weight:600;letter-spacing:1.6px;text-transform:uppercase;margin-bottom:6px}
.chapter h2{color:#e6edf3;font-size:24px;font-weight:700;letter-spacing:-0.4px;margin-bottom:8px;line-height:1.2}
.chapter h3{color:#e6edf3;font-size:16px;font-weight:600;margin:22px 0 8px}
.chapter h4{color:#c9d1d9;font-size:14px;font-weight:600;margin:16px 0 6px}
.chapter .lede{color:#8b949e;font-size:14px;margin-bottom:18px;max-width:760px;line-height:1.7}
.chapter p{color:#b0b8c4;margin-bottom:12px;max-width:800px}
.chapter ul{color:#b0b8c4;margin:8px 0 16px 24px}
.chapter li{margin-bottom:6px;line-height:1.7}
.chapter strong{color:#e6edf3;font-weight:600}
.card{background:#0d1117;border:1px solid #171d27;border-radius:12px;padding:20px;margin:12px 0}
.accent-l{border-left:3px solid #2ea043}
.accent-b{border-left:3px solid #1f6feb}
.accent-a{border-left:3px solid #bc8cff}
.accent-w{border-left:3px solid #d29922}
.accent-r{border-left:3px solid #f85149}
code{background:#161b22;color:#e6edf3;padding:2px 6px;border-radius:4px;font-family:ui-monospace,Menlo,monospace;font-size:12px}
pre{background:#161b22;border:1px solid #171d27;border-radius:8px;padding:14px 16px;overflow-x:auto;font-family:ui-monospace,Menlo,monospace;font-size:12px;color:#c9d1d9;line-height:1.5;margin:8px 0}
table.plain{width:100%;border-collapse:collapse;font-size:12px;margin:10px 0}
table.plain th{text-align:left;padding:8px 12px;color:#545d68;font-weight:600;text-transform:uppercase;font-size:10px;letter-spacing:0.8px;border-bottom:1px solid #171d27;background:#0d1117}
table.plain td{padding:8px 12px;border-bottom:1px solid #171d27;color:#c9d1d9;vertical-align:top}
table.plain td.mono{font-family:ui-monospace,Menlo,monospace}
table.plain tr:hover td{background:#0d1117}
.narr{color:#8b949e;font-size:13px;line-height:1.7;margin:10px 0;padding:10px 14px;border-left:2px solid #21262d}
.narr strong{color:#c9d1d9}
.ref{color:#545d68;font-size:11px;margin-top:6px;font-family:ui-monospace,Menlo,monospace}
.ref strong{color:#79c0ff;font-weight:600}
.step{display:flex;gap:14px;margin-bottom:14px;padding:12px 16px;background:#0d1117;border:1px solid #171d27;border-radius:8px}
.step .n{color:#58a6ff;font-weight:700;font-size:18px;flex-shrink:0;min-width:30px}
.step .body{color:#b0b8c4;font-size:13px;line-height:1.7}
.step .body strong{color:#e6edf3}
.footer{border-top:1px solid #171d27;padding:20px;text-align:center;color:#3d444d;font-size:11px}
@media(max-width:900px){
.layout{grid-template-columns:1fr}
.toc{display:none}
.wrap{padding:20px 14px 40px}
.bar nav{display:none}
}
</style></head>
<body>
<div class="bar">
<h1>Lakehouse — Technical Specification</h1>
<nav>
<a href=".">Dashboard</a>
<a href="console">Walkthrough</a>
<a href="proof">Architecture</a>
<a href="spec" class="active">Spec</a>
<a href="onboard">Onboard</a>
<a href="alerts">Alerts</a>
<a href="workspaces">Workspaces</a>
</nav>
<div class="rt">v1 · 2026-04-20</div>
</div>
<div class="layout">
<aside class="toc">
<div class="hdr">Contents</div>
<a href="#ch1">1. Repository layout</a>
<a href="#ch2">2. Data ingest pipeline</a>
<a href="#ch3">3. Measurement &amp; indexing</a>
<a href="#ch4">4. Contract inference</a>
<a href="#ch5">5. What a CRM can't do</a>
<a href="#ch6">6. How it gets better over time</a>
<a href="#ch7">7. Scale story — 20 staffers, 300 contracts, a surge</a>
<a href="#ch8">8. Error surfaces &amp; recovery</a>
<a href="#ch9">9. Per-staffer context</a>
<a href="#ch10">10. A day in the life</a>
<a href="#ch11">11. Known limits &amp; non-goals</a>
</aside>
<div class="wrap">
<!-- ═══ 1. REPO LAYOUT ═══ -->
<div class="chapter" id="ch1">
<div class="num">Chapter 1</div>
<h2>Repository layout</h2>
<div class="lede">What lives where. Every folder below has a single, bounded responsibility. A maintainer reading this should know — in under ten minutes — which crate owns a failing behavior.</div>
<table class="plain">
<thead><tr><th>Path</th><th>Owns</th></tr></thead>
<tbody>
<tr><td class="mono">crates/shared/</td><td>Types, errors, Arrow helpers, schema fingerprints, PII detection, secrets provider. Every other crate depends on this.</td></tr>
<tr><td class="mono">crates/storaged/</td><td>Raw object I/O. <code>BucketRegistry</code> (multi-bucket, rescue-aware), <code>AppendLog</code> (write-once batched append), <code>ErrorJournal</code> (bucket op failures). ADR-017 (federation), ADR-018 (append pattern).</td></tr>
<tr><td class="mono">crates/catalogd/</td><td>Metadata authority. Dataset manifests, schema fingerprints (ADR-020), tombstones (soft delete), AI-safe views, model profiles (Phase 17). In-memory index persisted as Parquet on storage.</td></tr>
<tr><td class="mono">crates/queryd/</td><td>SQL engine. DataFusion over Parquet + MemTable cache + delta merge-on-read + compaction. Registers every bucket as an object_store so SQL can join across them.</td></tr>
<tr><td class="mono">crates/ingestd/</td><td>Data on-ramp. CSV / JSON / PDF (+OCR via Tesseract) / Postgres streaming / MySQL streaming / inbox watcher / cron schedules. Every ingest path auto-tags PII (emails, phones, SSNs, addresses), records lineage, and marks embeddings stale.</td></tr>
<tr><td class="mono">crates/vectord/</td><td>The vector + learning surface. Embeddings stored as Parquet (ADR-008), HNSW index (Phase 15), trial system (autotune), promotion registry (Phase 16), playbook_memory (Phase 19). Core feedback loop lives here.</td></tr>
<tr><td class="mono">crates/vectord-lance/</td><td>Firewall crate. Lance 4.0 + Arrow 57, isolated from the main Arrow-55 workspace. Provides secondary vector backend for large-scale, random-access, and append-heavy workloads (ADR-019).</td></tr>
<tr><td class="mono">crates/journald/</td><td>Append-only mutation event log (ADR-012). Every insert/update/delete writes here — who, when, what, old/new value. Never mutated. Foundation for time-travel + compliance audit.</td></tr>
<tr><td class="mono">crates/aibridge/</td><td>Rust ↔ Python sidecar. HTTP client over FastAPI wrapper around Ollama. VRAM introspection via nvidia-smi. All LLM calls (embed, generate, rerank) flow through here.</td></tr>
<tr><td class="mono">crates/gateway/</td><td>Axum HTTP (:3100) + gRPC (:3101). Auth middleware, tools registry (Phase 12 — governed actions), CORS. Every external request enters here.</td></tr>
<tr><td class="mono">crates/ui/</td><td>Dioxus WASM developer UI. Internal tool. Not exposed externally.</td></tr>
<tr><td class="mono">mcp-server/</td><td>Bun/TypeScript recruiter-facing app. Serves <code>devop.live/lakehouse</code>. Routes: <code>/search /match /log /log_failure /clients/:c/blacklist /intelligence/* /memory/query /models/matrix /system/summary</code>. Observer sibling at <code>observer.ts</code> with HTTP listener on :3800 for scenario event ingest. Proxies to the Rust gateway for heavy work.</td></tr>
<tr><td class="mono">tests/multi-agent/</td><td>Dual-agent scenario harness + memory stack. <code>agent.ts</code> (prompts, continuation + tree-split primitives, cloud routing), <code>orchestrator.ts</code>, <code>scenario.ts</code> (contracts + staffer + tool_level), <code>kb.ts</code> (KB indexing, competence scoring, neighbor retrieval), <code>normalize.ts</code> (input normalizer — structured / regex / LLM), <code>memory_query.ts</code> (unified /memory/query), <code>gen_scenarios.ts</code> + <code>gen_staffer_demo.ts</code> (corpus generators), <code>run_e2e_rated.ts</code>, <code>chain_of_custody.ts</code>. Unit tests colocated (<code>kb.test.ts</code>, <code>normalize.test.ts</code>).</td></tr>
<tr><td class="mono">config/</td><td><code>models.json</code> — authoritative 5-tier model matrix (T1 hot local / T2 review local / T3 overview cloud / T4 strategic / T5 gatekeeper). Per-tier context_window + context_budget + overflow_policy. Read at runtime by scenario.ts; hot-swap friendly.</td></tr>
<tr><td class="mono">docs/</td><td><code>PRD.md</code>, <code>PHASES.md</code>, <code>DECISIONS.md</code> (20 ADRs). Every significant architectural choice has an ADR with the alternatives that were rejected and why.</td></tr>
<tr><td class="mono">data/</td><td>Default local object store. Parquet files per dataset, append-log batches, HNSW trial journals, promotion registries, <code>_playbook_memory/state.json</code> (now with retirement fields — Phase 25), catalog manifests. Plus four learning-loop directories: <code>_kb/</code> (signatures, outcomes, recommendations, error_corrections, config_snapshots, staffers), <code>_playbook_lessons/</code> (T3 cross-day lessons archived per run), <code>_observer/ops.jsonl</code> (append journal, durable scenario outcome stream), <code>_chunk_cache/</code> (spec'd for Phase 21 Rust port). Rebuildable from repo + this dir alone.</td></tr>
</tbody>
</table>
</div>
<!-- ═══ 2. INGEST PIPELINE ═══ -->
<div class="chapter" id="ch2">
<div class="num">Chapter 2</div>
<h2>Data ingest pipeline</h2>
<div class="lede">How staffing data gets into the system — whether from a CSV drop, an ATS export, a Postgres replica, or a PDF resume. Every path ends at the same place: a registered dataset with known schema, known lineage, known sensitivity.</div>
<div class="step"><div class="n">1</div><div class="body"><strong>Source arrives.</strong> Four shapes: (a) file upload via <code>POST /ingest/file</code>, (b) inbox watcher (drops in <code>./inbox/</code> → auto-ingested in under 15s), (c) Postgres or MySQL streaming connector (<code>POST /ingest/db</code> with DSN), (d) scheduled ingest via <code>ingestd::schedule</code> with cron.</div></div>
<div class="step"><div class="n">2</div><div class="body"><strong>Parse + normalize.</strong> CSV parser infers types per column; defaults to <code>String</code> on ambiguity (ADR-010 — better to ingest everything than reject on type mismatch). JSON parser flattens nested objects. PDF extractor uses <code>lopdf</code> first; falls back to Tesseract OCR for scanned/image PDFs. Output is always an Arrow <code>RecordBatch</code>.</div></div>
<div class="step"><div class="n">3</div><div class="body"><strong>Auto-detect PII.</strong> <code>shared::pii</code> scans column values and names. Identifies emails, phone numbers, SSNs, salaries, street addresses, medical terms. Tags columns with <code>sensitivity: PII | PHI | Financial | Internal | Public</code> (Phase 10 catalog v2).</div></div>
<div class="step"><div class="n">4</div><div class="body"><strong>Deduplicate by content hash.</strong> Every uploaded file's SHA-256 is checked against the catalog's seen-hash log. Re-ingesting the same file is a no-op (ADR invariant #5).</div></div>
<div class="step"><div class="n">5</div><div class="body"><strong>Write Parquet to object storage.</strong> <code>arrow_helpers::record_batch_to_parquet</code> → <code>storaged::ops::put</code> → file lands under <code>data/datasets/&lt;name&gt;.parquet</code> (or bucket-scoped via <code>BucketRegistry</code>). Schema fingerprint computed.</div></div>
<div class="step"><div class="n">6</div><div class="body"><strong>Register in catalog.</strong> <code>catalogd::Registry::register(name, fingerprint, objects)</code> — idempotent on (name, fingerprint). Same name + same fingerprint = reuse manifest, bump updated_at. Same name + different fingerprint = <code>409 Conflict</code> (ADR-020 — prevents silent schema drift). New name = create new manifest with owner, lineage, freshness SLA, column metadata, PII tags.</div></div>
<div class="step"><div class="n">7</div><div class="body"><strong>Mark embeddings stale.</strong> If the dataset already has a vector index, the new rows mean that index is now behind. <code>Registry::mark_embeddings_stale</code> flips a flag; <code>POST /vectors/refresh/&lt;dataset&gt;</code> runs an incremental re-embed (only new rows, not the whole corpus).</div></div>
<div class="step"><div class="n">8</div><div class="body"><strong>Queryable immediately.</strong> <code>queryd::context</code> picks up the new manifest on next query. Hot-cache warms on first hit. Delta merge-on-read means updates land without rewriting the base Parquet.</div></div>
<div class="ref"><strong>Code:</strong> crates/ingestd/src/{service.rs, csv.rs, json.rs, pdf.rs, pg_stream.rs, my_stream.rs, schedule.rs}</div>
</div>
<!-- ═══ 3. MEASUREMENT & INDEXING ═══ -->
<div class="chapter" id="ch3">
<div class="num">Chapter 3</div>
<h2>Measurement &amp; indexing</h2>
<div class="lede">Once data is in, the system describes it rigorously and builds fast-access indexes over the parts that will be queried. Every measurement is deterministic, versioned, and visible via HTTP.</div>
<h3>What gets measured per dataset</h3>
<ul>
<li><strong>Row count</strong> (from parquet footer, not a SELECT COUNT). O(1).</li>
<li><strong>Schema fingerprint</strong> — SHA-256 over (column_name, type, nullability, sort) tuples. Drives ADR-020 idempotent register.</li>
<li><strong>Owner / sensitivity / freshness SLA</strong> — catalog v2 metadata. PII auto-detected; owner assigned on ingest.</li>
<li><strong>Lineage</strong> — source_system → ingest_job → dataset. Who put this here, when, from what.</li>
<li><strong>Last embedded at</strong> — when the vector index covering this dataset was last refreshed. Drives stale-detection.</li>
</ul>
<h3>How vector indexes are built</h3>
<p>Two backends, chosen per profile (ADR-019):</p>
<table class="plain">
<thead><tr><th></th><th>HNSW over Parquet (primary)</th><th>Lance (secondary)</th></tr></thead>
<tbody>
<tr><td>Storage</td><td>Embeddings as Parquet columns (<code>doc_id, chunk_text, vector</code>)</td><td>Lance native dataset</td></tr>
<tr><td>Index</td><td>HNSW in RAM, serialized sidecar</td><td>IVF_PQ on disk</td></tr>
<tr><td>Build time (100K × 768d)</td><td>~230s</td><td>~16s (14× faster)</td></tr>
<tr><td>Search p50 (100K)</td><td>~873μs</td><td>~7.4ms at recall 1.0</td></tr>
<tr><td>Append</td><td>Rewrite required</td><td>Structural (0.08s for 100 rows)</td></tr>
<tr><td>Random fetch by doc_id</td><td>Full scan</td><td>~311μs (112× faster)</td></tr>
<tr><td>RAM ceiling</td><td>~5M vectors</td><td>Scales past RAM — disk-resident</td></tr>
</tbody>
</table>
<h3>Autotune</h3>
<p>The <code>vectord::agent</code> background task runs continuously. Per index, it proposes HNSW configurations (<code>ef_construction × ef_search</code>), executes a trial against a stored eval set, journals the result as JSONL, and — if recall beats the min_recall gate (0.9) and latency wins the Pareto test — promotes the new config atomically via <code>promotion_registry</code>. No downtime. Rollback in milliseconds.</p>
<h3>Per-profile / per-staffer indexing</h3>
<p>Model profiles (Phase 17) are not routing strings — they are named scopes. Each profile has <code>bound_datasets[]</code>, <code>hnsw_config</code>, <code>vector_backend</code>, and <code>bucket</code>. When a staffer activates a profile:</p>
<ul>
<li>EmbeddingCache warms for bound indexes only</li>
<li>HNSW is rebuilt with the profile's config (if different from current)</li>
<li>Search via <code>POST /vectors/profile/&lt;id&gt;/search</code> rejects out-of-scope queries with 403 + list of allowed bindings</li>
<li>Ollama swaps to the profile's model via <code>keep_alive=0</code>; only one model in VRAM at a time</li>
</ul>
<h3>Model matrix (Phase 20)</h3>
<p>Five tiers declared in <code>config/models.json</code>. Each call site picks the tier appropriate to its purpose — hot-path JSON emitters get fast local, overview/strategic/gatekeeper decisions get thinking models on cloud. Every tier carries <code>context_window</code>, <code>context_budget</code>, and <code>overflow_policy</code>.</p>
<table class="plain">
<thead><tr><th>Tier</th><th>Purpose</th><th>Primary model</th><th>Frequency</th></tr></thead>
<tbody>
<tr><td>T1 hot</td><td>Per tool call — SQL gen, hybrid_search, propose_done</td><td><code>qwen3.5:latest</code> local, <code>think:false</code></td><td>50-200/scenario</td></tr>
<tr><td>T2 review</td><td>Per-step consensus, drift flagging</td><td><code>qwen3:latest</code> local, <code>think:false</code></td><td>5-14/event</td></tr>
<tr><td>T3 overview</td><td>Mid-day checkpoints + cross-day lesson distill</td><td><code>gpt-oss:120b</code> Ollama Cloud, thinking on</td><td>1-3/scenario</td></tr>
<tr><td>T4 strategic</td><td>Pattern re-ranking, weekly gap audit</td><td><code>qwen3.5:397b</code> cloud</td><td>1-10/day</td></tr>
<tr><td>T5 gatekeeper</td><td>Schema migrations, autotune config changes</td><td><code>kimi-k2-thinking</code> cloud, audit-logged</td><td>1-5/day</td></tr>
</tbody>
</table>
<p><strong>Key mechanical finding (2026-04-21):</strong> qwen3.5 and qwen3 are <em>thinking</em> models — they burn ~650 tokens of hidden reasoning before emitting the visible response. For hot-path JSON emitters this meant 400-token budgets returned empty strings. Fix: <code>think: false</code> plumbed through sidecar's <code>/generate</code> endpoint; hot path disables thinking (structure matters more than reasoning depth), overseer tiers keep it on. Mistral was dropped entirely after a 0/14 fill rate on complex scenarios (decoder-level malformed-JSON bug, not a prompt issue).</p>
<p><strong>Continuation primitive (Phase 21):</strong> <code>generateContinuable()</code> handles output-overflow without <code>max_tokens</code> tourniquets — empty response → geometric backoff retry; truncated-JSON → continue with partial as scratchpad. <code>generateTreeSplit()</code> handles input-overflow via map-reduce with running scratchpad. Both respect <code>assertContextBudget()</code> so silent truncation can't happen.</p>
<h3>Per-staffer tool_level (Phase 23)</h3>
<p>Scenarios can be scoped to a specific coordinator (<code>staffer: {id, name, tenure_months, role, tool_level}</code>). <code>tool_level</code> controls which tiers are available:</p>
<ul>
<li><code>full</code> — T1/T2 local, T3 cloud, cloud rescue on failure</li>
<li><code>local</code> — T1/T2/T3 all local (gpt-oss:20b as overseer)</li>
<li><code>basic</code> — kimi-k2.5 cloud executor + local reviewer + local T3, no rescue</li>
<li><code>minimal</code> — kimi-k2.5 cloud executor, no T3, no rescue. Playbook inheritance is the only signal.</li>
</ul>
<p>Measured 2026-04-21 on a 36-run demo (4 staffers × 3 contracts × 3 rounds): James Park (mid, local tools) ranked first at 92.9% fill and 36.8 cites/run, Maria Chen (senior, full tools) second at 81.0%. Cloud T3 adds latency without measurable benefit on this workload. Alex Rivera (trainee, minimal) still hit 59.5% fill purely from playbook inheritance — proof the memory carries knowledge when the model is capable.</p>
<div class="ref"><strong>Code:</strong> crates/vectord/src/{hnsw.rs, autotune.rs, agent.rs, promotion.rs} · tests/multi-agent/{agent.ts, scenario.ts} · config/models.json · ADR-019</div>
</div>
<!-- ═══ 4. CONTRACT INFERENCE ═══ -->
<div class="chapter" id="ch4">
<div class="num">Chapter 4</div>
<h2>Contract inference from external signal</h2>
<div class="lede">Most CRMs wait for a contract to land. This system watches upstream demand and pre-builds the ranking before the contract lands.</div>
<p>The concrete example running on <code>devop.live/lakehouse</code> is Chicago Department of Buildings permit data (public Socrata API). Every permit is a signal that construction — and therefore staffing — is coming.</p>
<h3>Flow</h3>
<div class="step"><div class="n">1</div><div class="body"><strong>Fetch.</strong> <code>/intelligence/market</code> and <code>/intelligence/permit_contracts</code> hit <code>data.cityofchicago.org/resource/ydr8-5enu.json</code> live. No caching of permit data — every page load is fresh.</div></div>
<div class="step"><div class="n">2</div><div class="body"><strong>Map work_type → role.</strong> Industry dictionary: "Electrical Work" → "Electrician", "Masonry Work" → "Production Worker", "Mechanical Work" → "Maintenance Tech", etc.</div></div>
<div class="step"><div class="n">3</div><div class="body"><strong>Derive worker count.</strong> Heuristic: ~1 worker per $150K of permit cost, capped 2-8 per contract for staffing realism. Operator-configurable when real client history is available.</div></div>
<div class="step"><div class="n">4</div><div class="body"><strong>Derive timeline.</strong> Permit issued → construction starts ~45 days later → staffing window opens ~14 days before construction. Classifies each permit as <code>overdue</code>, <code>urgent</code>, <code>soon</code>, <code>scheduled</code>.</div></div>
<div class="step"><div class="n">5</div><div class="body"><strong>Run hybrid search against the bench.</strong> For each derived contract, <code>POST /vectors/hybrid</code> with <code>sql_filter</code> on role+state+city+availability, <code>use_playbook_memory: true</code>, <code>playbook_memory_k: 200</code>. Returns top-5 candidates with boost + citations.</div></div>
<div class="step"><div class="n">6</div><div class="body"><strong>Query the meta-index.</strong> <code>POST /vectors/playbook_memory/patterns</code> aggregates traits across similar past playbooks — recurring certs, skills, archetype, reliability distribution. Surfaces signal the operator didn't query for.</div></div>
<div class="step"><div class="n">7</div><div class="body"><strong>Render on the dashboard.</strong> Each card shows permit + derived contract + top 3 candidates with memory chips + discovered pattern + urgency. All of this pre-computed before any staffer opens the UI.</div></div>
<h3>Coverage forecast</h3>
<p><code>/intelligence/staffing_forecast</code> aggregates the last 30 days of permits into predicted role-level demand, joins against the IL bench supply, computes coverage %, and classifies each role as <code>critical</code> / <code>tight</code> / <code>watch</code> / <code>ok</code>. The dashboard's top panel renders this — staffers see supply gaps before they query.</p>
</div>
<!-- ═══ 5. WHAT A CRM CAN'T DO ═══ -->
<div class="chapter" id="ch5">
<div class="num">Chapter 5</div>
<h2>What a CRM can't do (and why)</h2>
<div class="lede">A CRM stores. This system infers, predicts, re-ranks, and compounds. The six capabilities below are load-bearing — missing any of them is the gap between "software that logs calls" and "software that makes the next call better."</div>
<div class="card accent-b">
<table class="plain">
<thead><tr><th>Capability</th><th>CRM</th><th>This system</th></tr></thead>
<tbody>
<tr><td>Store candidate records</td><td>Yes</td><td>Yes (workers_500k, candidates)</td></tr>
<tr><td>Search by structured field</td><td>Yes</td><td>Yes (DataFusion SQL, sub-100ms on 3M rows)</td></tr>
<tr><td>Search by semantic meaning</td><td>No</td><td>Yes (HNSW + nomic-embed-text)</td></tr>
<tr><td>Combine SQL filter + semantic rank</td><td>No</td><td>Yes (<code>/vectors/hybrid</code>)</td></tr>
<tr><td>Boost workers based on past success</td><td>No</td><td>Yes (Phase 19 playbook_memory)</td></tr>
<tr><td>Penalize workers based on past failure</td><td>No</td><td>Yes (<code>/log_failure</code> + <code>0.5<sup>n</sup></code> penalty)</td></tr>
<tr><td>Surface traits across past fills</td><td>No</td><td>Yes (<code>/vectors/playbook_memory/patterns</code>)</td></tr>
<tr><td>Predict staffing demand from external data</td><td>No</td><td>Yes (Chicago permit feed + 30-day rolling forecast)</td></tr>
<tr><td>Count down to staffing deadline per contract</td><td>No</td><td>Yes (permit issue_date + heuristic timeline)</td></tr>
<tr><td>Explain why each candidate ranked</td><td>No</td><td>Yes (boost chip + narrative citations + memory pattern)</td></tr>
<tr><td>Improve ranking from operator actions</td><td>No</td><td>Yes (every Call/SMS/No-show click → re-rank signal)</td></tr>
</tbody>
</table>
</div>
</div>
<!-- ═══ 6. HOW IT GETS BETTER ═══ -->
<div class="chapter" id="ch6">
<div class="num">Chapter 6</div>
<h2>How it gets better over time</h2>
<div class="lede">Compounding learning across seven paths. The first three are automatic background loops. Paths 4-7 landed 2026-04-21 and turn the system into a reinforcement-learning pipeline: outcomes → knowledge base → pathway recommendations → cloud rescue → competence-weighted retrieval → observer analysis. All seven happen without operator intervention.</div>
<h3>Path 1 — Playbook boost with geo + role prefilter (Phase 19 + refinement)</h3>
<p>Every sealed fill is seeded to <code>playbook_memory</code>. The boost fires inside <code>/vectors/hybrid</code> when <code>use_playbook_memory: true</code>. Math, tightened 2026-04-21 after a diagnostic pass found globally-ranked playbooks were missing the SQL-filtered candidate pool entirely:</p>
<pre>per_worker = cosine(query_emb, playbook_emb) × 0.5 × e^(-age/30) × 0.5^failures / n_workers
boost[(city, state, name)] = min(Σ per_worker, 0.25)</pre>
<p><strong>Multi-strategy retrieval (new):</strong> before cosine, <code>compute_boost_for_filtered_with_role(target_geo, target_role)</code> prefilters to same-city playbooks, then gives exact (role, city, state) matches similarity=1.0 and fills up to half the top-k. Cosine fills the rest. Mirrors 2026 Mem0/Zep guidance on parallel-strategy rerank.</p>
<p><strong>Measured lift:</strong> before geo-filter, Nashville Welder query returned <code>boosts=170 matched=0</code> (zero intersection with candidate pool). After: <code>boosts=36 matched=11</code>. On the Riverfront Steel scenario, total playbook citations went from 2 → 28 per run — a 14× delta on identical inputs. The diagnostic log <code>playbook_boost: boosts=N sources=N parsed=N matched=N target_geo=? target_role=?</code> runs on every hybrid call so the class of silent-miss bug stays visible.</p>
<h3>Path 2 — Pattern discovery (meta-index)</h3>
<p><code>/vectors/playbook_memory/patterns</code> goes beyond "who was endorsed" to answer "what did past similar fills have in common?" Aggregates recurring certifications, skills, archetype, reliability distribution across the top-K semantically similar playbooks. Surfaces signal the operator didn't explicitly query for.</p>
<h3>Path 3 — Autotune agent</h3>
<p>The <code>vectord::agent</code> background task runs continuously. Watches the HNSW trial journal, proposes configs, executes trials, promotes Pareto winners — without human intervention. Operator sees "the index got faster overnight" and doesn't know why. The journal knows why.</p>
<h3>Path 4 — Knowledge Base + pathway recommender (Phase 22)</h3>
<p>Meta-layer over playbook_memory. Files under <code>data/_kb/</code>:</p>
<ul>
<li><code>signatures.jsonl</code> — sig_hash + embedding of every run's event shape</li>
<li><code>outcomes.jsonl</code> — per-run summary (models, fill rate, turns, citations, rescue stats, staffer, elapsed)</li>
<li><code>pathway_recommendations.jsonl</code> — AI-synthesized advice for the next run of a similar sig</li>
<li><code>error_corrections.jsonl</code> — fail→succeed deltas on the same sig</li>
<li><code>config_snapshots.jsonl</code> — hash of active models + tool_level at each run</li>
</ul>
<p>Cycle: scenario ends → <code>kb.indexRun()</code> appends outcome → <code>kb.recommendFor(nextSpec)</code> finds k-NN signatures, feeds outcome history to an overview model, writes structured JSON advice → next scenario reads it via <code>kb.loadRecommendation(spec)</code> and injects <code>pathway_notes</code> into the executor's context alongside prior T3 lessons.</p>
<h3>Path 5 — Cloud rescue on failure (Phase 22 item B)</h3>
<p>When an event fails (drift abort, JSON parse, pool exhaustion) and cloud T3 is enabled, <code>requestCloudRemediation()</code> feeds the full failure trace — SQL filters attempted, row counts, reviewer drift notes, gap signals, contract terms — to <code>gpt-oss:120b</code> on Ollama Cloud. Cloud returns structured <code>{retry, new_city, new_state, new_role, new_count, rationale}</code>. Event retries once with the pivot. Verified on stress_01: Gary IN (zero workers indexed) misplacement → cloud proposed South Bend IN → retry filled 1/1.</p>
<h3>Path 6 — Staffer competence-weighted retrieval (Phase 23)</h3>
<p>Answers "who handled this" as a first-class matrix-index dimension. Each scenario carries <code>staffer: {id, name, tenure_months, role, tool_level}</code>. After every run, <code>recomputeStafferStats(staffer_id)</code> aggregates their fill_rate, turn efficiency, citation density, rescue rate into a single <code>competence_score</code> (0.45·fill + 0.20·turn_eff + 0.20·cites + 0.15·rescue).</p>
<p><code>findNeighbors</code> returns <code>weighted_score = cosine × max_staffer_competence</code> — top-performer playbooks rank above juniors' on similar scenarios. Auto-discovery emerges: running 4 staffers × 3 contracts × 3 rounds surfaced Rachel D. Lewis (Welder Nashville) with 18 endorsements across all 4 staffers, Angela U. Ward (Machine Op Indianapolis) with 19 — reliable-performer labels the system built without human tagging.</p>
<h3>Path 7 — Observer outcome ingest (Phase 24)</h3>
<p>Observer runs as <code>lakehouse-observer.service</code>, now with an HTTP listener on <code>:3800</code>. Scenarios POST per-event outcomes to <code>/event</code> with full provenance (staffer_id, sig_hash, event_kind, role, city, state, rescue flags). Observer's ERROR_ANALYZER and PLAYBOOK_BUILDER loops consume them alongside MCP-wrapped ops. Persistence switched from the old <code>/ingest/file</code> REPLACE path to an append-only <code>data/_observer/ops.jsonl</code> journal so the trace survives across restarts.</p>
<h3>Input normalizer + unified memory query</h3>
<p>Two surfaces added 2026-04-21 to make the memory stack respond coherently to any input shape:</p>
<ul>
<li><code>normalizeInput(raw)</code> — accepts structured JSON, natural language, or mixed. Three-tier: structured fast-path → regex (handles "need 3 welders in Nashville, TN" in 0ms without an LLM call) → qwen3 LLM fallback for low-signal inputs.</li>
<li><code>POST /memory/query</code> — one endpoint, returns every memory surface in a single bundle: <code>playbook_workers</code> (with boost + citations), <code>pathway_recommendation</code> (KB), <code>neighbor_signatures</code> (competence-weighted), <code>prior_lessons</code> (T3 overseer history), <code>top_staffers</code> (leaderboard), <code>discovered_patterns</code> (auto-surfaced reliable workers for this role+city), <code>latency_ms</code> (per-source). Natural-language query end-to-end: 319ms.</li>
</ul>
<h3>Honest gaps — what we can still implement</h3>
<p>Three of the five 2026-era memory findings remain unwired. Flagged for near-term implementation, not hidden:</p>
<ul>
<li><strong>Zep-style validity windows</strong> — playbook entries have timestamps but no <code>valid_until</code> or <code>schema_fingerprint</code>. Load-bearing: when a schema migration changes a column, stale playbooks silently keep boosting. Biggest-value remaining fix.</li>
<li><strong>Mem0-style UPDATE / DELETE / NOOP ops</strong> — <code>/seed</code> only ADDs. Same (operation, date) pair appends a duplicate instead of refining an existing entry. Playbook file grows faster than necessary.</li>
<li><strong>Letta working-memory hot cache</strong> — every query scans all 1560 playbook entries from disk. Cheap today at 1.5K, not at 100K. Solution: LRU in-process cache for the last N playbooks or the current sig's neighborhood.</li>
</ul>
<p>Validity windows is next — preserves the trust signal (boost only fires on playbooks that are still true given the current schema) rather than the latency signal (which the current scale doesn't need yet).</p>
<div class="ref"><strong>Code:</strong> crates/vectord/src/{playbook_memory.rs, service.rs} · tests/multi-agent/{kb.ts, memory_query.ts, normalize.ts, scenario.ts} · mcp-server/{observer.ts, index.ts} · data/_kb/ · data/_observer/</div>
</div>
<!-- ═══ 7. SCALE STORY ═══ -->
<div class="chapter" id="ch7">
<div class="num">Chapter 7</div>
<h2>Scale story — 20 staffers, 300 contracts, a surge</h2>
<div class="lede">What happens when the demo-level load becomes the production-level load, and midday a client pushes 20 more contracts plus a 1M-row ATS delta. Honest: some of this is architectural headroom, not measured scale. The designed behaviors are below.</div>
<h3>20 concurrent staffers</h3>
<p><strong>Axum is async.</strong> The gateway handles concurrent requests on Tokio with work-stealing. No per-request thread. Tested at 10 parallel queries in 82ms total on this hardware.</p>
<p><strong>Per-staffer profile isolation.</strong> Each staffer activates their own profile (Phase 17) or workspace (Phase 8.5). Profile scopes their search to bound datasets. Workspace carries their in-progress contracts across sessions.</p>
<p><strong>Per-client blacklists.</strong> Auto-applied when the caller passes <code>client: "X"</code> on <code>/search</code>. Staffer A filling for Acme never sees Acme's flagged workers. Staffer B filling for MidState sees them normally.</p>
<h3>300 active contracts</h3>
<p><strong>SQL on <code>job_orders</code> is cheap.</strong> 300 rows is nothing — a scan is microseconds.</p>
<p><strong>Workspace per contract.</strong> Each contract gets its own workspace with saved searches, shortlists, activity log. Zero-copy handoff between staffers (pointer swap, not data copy).</p>
<p><strong>Forecast remains coherent.</strong> <code>/intelligence/staffing_forecast</code> aggregates 30-day permit data regardless of contract count. The bench supply query (<code>GROUP BY role</code> over workers_500k) is a single sub-second SQL.</p>
<h3>Midday surge: +20 contracts, +1M profiles</h3>
<p>The delta arrives at 12:30. Here's what happens in the following minutes:</p>
<div class="step"><div class="n">1</div><div class="body"><strong>+20 contracts via /ingest/db or /ingest/file.</strong> Parsed, schema-checked, Parquet-written, catalog-registered. No queries blocked — register holds a write lock across the manifest write only.</div></div>
<div class="step"><div class="n">2</div><div class="body"><strong>+1M worker profiles arrives as delta to workers_500k.</strong> Append-log pattern (ADR-018) means the new rows write to a fresh batch file — base Parquet is NOT rewritten. Queries against workers_500k immediately merge-on-read the new batches.</div></div>
<div class="step"><div class="n">3</div><div class="body"><strong>Embeddings marked stale.</strong> The vector index for workers_500k_v1 now has 1M rows it hasn't seen. <code>mark_embeddings_stale</code> flips the flag.</div></div>
<div class="step"><div class="n">4</div><div class="body"><strong>Incremental refresh fires.</strong> <code>POST /vectors/refresh/workers_500k</code> reads only the new rows (diff against existing embeddings), embeds them in batches of 64 via Ollama, writes delta embedding Parquet. Measured on threat_intel: 34 new rows in 970ms (6× faster than full re-embed).</div></div>
<div class="step"><div class="n">5</div><div class="body"><strong>Search degrades gracefully.</strong> During the refresh, searches against workers_500k_v1 still work — they serve from the old embeddings. Brute-force cosine over new-rows-without-embeddings is allowed but costs more. HNSW rebuild happens after all embeds complete.</div></div>
<div class="step"><div class="n">6</div><div class="body"><strong>Hot-swap promotion.</strong> When the new index is ready, <code>promotion_registry</code> atomically flips the active pointer. Next search hits the new config. Rollback stays available.</div></div>
<div class="step"><div class="n">7</div><div class="body"><strong>Autotune re-enters the loop.</strong> The agent queue picks up a <code>DatasetAppended</code> trigger and schedules a fresh HNSW trial cycle against the expanded index.</div></div>
<h3>Known pain points at this scale</h3>
<ul>
<li><strong>Ollama inference is serial.</strong> Embedding 1M rows at ~50 chunks/sec through nomic-embed-text = ~6 hours. Acceptable for overnight refresh, not for "immediate." Mitigated by incremental refresh (only deltas).</li>
<li><strong>RAM ceiling on HNSW.</strong> Around 5M vectors × 768d, HNSW stops fitting in 128GB comfortably. Mitigation: per-profile <code>vector_backend: lance</code> flip — disk-resident IVF_PQ scales past the RAM line (ADR-019).</li>
<li><strong>VRAM ceiling for model variety.</strong> A4000 16GB holds 1-2 loaded models. Multi-model recruiter surfaces are a sequential swap, not parallel (Ollama <code>keep_alive=0</code>). Phase 17 profile activation unloads the prior model on swap.</li>
<li><strong>playbook_memory growth.</strong> 1936 entries today. Phase 25 (2026-04-21) added retirement via <code>valid_until</code> + <code>schema_fingerprint</code> fields + <code>POST /vectors/playbook_memory/retire</code> endpoint (manual or schema-drift triggered). Active vs retired split surfaced on <code>GET /vectors/playbook_memory/status</code>. Brute-force cosine still sub-ms at current size; Letta-style working-memory hot cache deferred until entry count crosses ~100K.</li>
</ul>
</div>
<!-- ═══ 8. ERROR SURFACES ═══ -->
<div class="chapter" id="ch8">
<div class="num">Chapter 8</div>
<h2>Error surfaces &amp; recovery</h2>
<div class="lede">Every failure mode has a named surface, a structured response, and a recovery path. No silent failures.</div>
<table class="plain">
<thead><tr><th>Failure mode</th><th>Surface / response</th><th>Recovery</th></tr></thead>
<tbody>
<tr><td>Ingest receives file with schema mismatch vs existing dataset</td><td><code>409 Conflict</code> with both fingerprints named (ADR-020)</td><td>Re-ingest under a new name, or migrate the existing via Phase 14 schema evolution</td></tr>
<tr><td>Bucket unreachable on write</td><td>Hard 503, error journaled to <code>primary://_errors/bucket_errors/</code></td><td><code>GET /storage/errors</code> lists failures; <code>GET /storage/bucket-health</code> shows per-bucket status</td></tr>
<tr><td>Bucket unreachable on read</td><td>Rescue bucket fallback, <code>X-Lakehouse-Rescue-Used: true</code> header on response</td><td>Response still succeeds; operator sees rescue flag</td></tr>
<tr><td>/log receives name that doesn't exist in workers_500k</td><td>Seed is SKIPPED; response includes <code>rejected_ghost_names: [...]</code> and a note</td><td>Operator sees exactly which names were rejected and why</td></tr>
<tr><td>Dual-agent executor malforms tool call</td><td>Result appended to log with <code>error</code> field; counter increments</td><td>After 3 consecutive: abort with full log dump at <code>tests/multi-agent/playbooks/&lt;id&gt;-FAILED.json</code></td></tr>
<tr><td>Dual-agent drifts from target</td><td>Reviewer verdict = <code>drift</code>, counter increments</td><td>After 3 consecutive drifts: abort with full log</td></tr>
<tr><td>Hybrid search finds zero candidates</td><td>Returns empty <code>sources[]</code> + <code>sql_matches: 0</code></td><td>Gap signal captured by scenario runner; operator prompted to broaden filter</td></tr>
<tr><td>Ollama sidecar down</td><td>502 Bad Gateway from aibridge; <code>embed</code> calls fail fast</td><td>Restart: <code>systemctl restart lakehouse-sidecar</code>; vector search falls back to pre-computed embeddings</td></tr>
<tr><td>Gateway restart mid-operation</td><td>In-memory state (playbook_memory, HNSW) reloaded from persisted <code>state.json</code> / trial journals</td><td>Zero data loss; catalog, storage, journals are all source-of-truth</td></tr>
<tr><td>Schema fingerprint diverges across manifests</td><td><code>catalog::dedupe</code> reports <code>DedupeReport</code> with winner selection (non-null row_count first, then newest updated_at)</td><td><code>POST /catalog/dedupe</code> collapses duplicates idempotently</td></tr>
<tr><td>Scenario event fails on zero-supply city</td><td>Cloud rescue (Phase 22 item B) fires — <code>gpt-oss:120b</code> sees SQL filters attempted, row counts, reviewer drift notes, contract terms; returns structured <code>{retry, new_city, new_state, new_role, new_count, rationale}</code></td><td>Retry with pivot runs same executor loop with new geography; verified Gary IN → South Bend IN filled 1/1 after original drift-abort</td></tr>
<tr><td>LLM response truncated mid-JSON (thinking model ate token budget)</td><td>Phase 21 <code>generateContinuable()</code> detects via brace-balance + JSON.parse; no silent truncation</td><td>Auto-continue with partial as scratchpad, or geometric backoff if initial call returned empty. Bounded by <code>max_continuations</code>.</td></tr>
<tr><td>Schema migration invalidates existing playbooks</td><td>Phase 25 — <code>POST /vectors/playbook_memory/retire</code> with current fingerprint retires all mismatched entries in scope; diagnostic log shows counts</td><td>Retired entries stay in journal for forensics but are skipped by all boost calculations. Scoped by (city, state) so unrelated geos aren't touched.</td></tr>
<tr><td>Observer fails to reach scenario outcome stream</td><td>Scenario <code>postObserverEvent()</code> uses 2s <code>AbortSignal.timeout</code>; silent skip if :3800 is down</td><td>Scenario log is still the source of truth; observer re-ingest on next run restores the stream. <code>data/_observer/ops.jsonl</code> is append-only so prior events survive.</td></tr>
</tbody>
</table>
</div>
<!-- ═══ 9. PER-STAFFER CONTEXT ═══ -->
<div class="chapter" id="ch9">
<div class="num">Chapter 9</div>
<h2>Per-staffer context</h2>
<div class="lede">Twenty staffers don't see the same UI state. Each one's session is shaped by their active profile, their workspaces, their assigned contracts, and their client's blacklists.</div>
<h3>Active profile (Phase 17)</h3>
<p>Scopes every search. A <code>staffing-recruiter</code> profile bound to <code>workers_500k</code> sees only that dataset. A <code>security-analyst</code> profile bound to <code>threat_intel</code> cannot see worker data. <code>GET /vectors/profile/&lt;id&gt;/audit</code> records every tool invocation by model identity.</p>
<h3>Workspace (Phase 8.5)</h3>
<p>Per-contract state. Each workspace has daily/weekly/monthly tiers, saved searches, shortlists, activity logs. Survives across sessions. Instant zero-copy handoff between staffers — pointer swap, not data copy. Persisted to object storage, rebuilt on startup.</p>
<h3>Client blacklist</h3>
<p>Per-client worker exclusion. Populated via <code>POST /clients/:client/blacklist</code>. Auto-applied when the caller passes <code>client: "X"</code> on <code>/search</code>. JSON-backed; would move to catalog table under real client load.</p>
<h3>Audit trail</h3>
<p>Phase 12 tool registry logs every governed-action invocation (who called what, with what args, when, outcome). <code>GET /tools/audit</code> queryable. Phase 13 access control layers on top — role-based field masking, query audit log.</p>
<h3>Daily summary per staffer</h3>
<p>Workspace activity log + per-staffer filter on the event journal gives <strong>"what did Sarah do today"</strong> as a direct query. The foundation for shift-handoff reports.</p>
<h3>Staffer identity + competence-weighted retrieval (Phase 23)</h3>
<p>Each scenario run carries an explicit <code>staffer: {id, name, tenure_months, role, tool_level}</code>. The KB aggregates per-staffer stats (<code>data/_kb/staffers.jsonl</code>) that roll up into a single <code>competence_score</code>:</p>
<pre>competence_score = 0.45·fill_rate + 0.20·turn_efficiency + 0.20·citation_density + 0.15·rescue_rate</pre>
<p>When any query runs <code>kb.findNeighbors(spec, k)</code>, the ranking isn't just cosine similarity — it's <code>weighted_score = cosine × max_staffer_competence</code> over the best coordinator who ran that signature. Senior staffers' playbooks surface above juniors' on similar scenarios, even when the juniors' scenario was marginally closer in embedding space.</p>
<p>The <code>tool_level</code> knob (full / local / basic / minimal) controls which tiers are available to a given staffer's runs. See Ch 3 for the mapping. Variance is real and measurable: 36-run demo produced a 46pt fill-rate delta between James (local tools, 93%) and Alex (minimal tools, 60%) on identical contracts.</p>
<h3>Auto-discovered reliable-performer labels</h3>
<p>A second-order output of the competence path: when multiple staffers independently endorse the same worker on similar-signature playbooks, that worker accumulates cross-staffer endorsements. <code>scripts/kb_staffer_report.py</code> surfaces them — after 36 runs, Rachel D. Lewis (Welder Nashville) had 18 endorsements across 4 staffers, Angela U. Ward (Machine Op Indianapolis) 19. These are high-confidence "reliable" labels the system produced without human tagging. The UI could badge these workers on future queries; today they're visible via <code>/memory/query</code>'s <code>discovered_patterns</code> bundle.</p>
</div>
<!-- ═══ 10. A DAY IN THE LIFE ═══ -->
<div class="chapter" id="ch10">
<div class="num">Chapter 10</div>
<h2>A day in the life — from morning brief to EOD retrospective</h2>
<div class="lede">Concrete operator timeline. Every step touches a real endpoint that exists today.</div>
<div class="step"><div class="n">07:00</div><div class="body"><strong>Overnight housekeeping.</strong> Scheduled ingest runs — the configured cron picks up the client's latest ATS CSV delta, runs it through the pipeline in Ch2, marks workers_500k embeddings stale. Autotune agent promotes any Pareto-winner HNSW configs from overnight trials.</div></div>
<div class="step"><div class="n">07:30</div><div class="body"><strong>Embedding refresh.</strong> Background job re-embeds the new rows. Old index keeps serving. Hot-swap promotes when done.</div></div>
<div class="step"><div class="n">08:00</div><div class="body"><strong>Sarah (staffer) opens devop.live/lakehouse.</strong> Page loads in ~3s. Forecast panel shows: "$275M construction coming, 4 tight roles this week." Live Contracts section shows 6 Chicago permits with proposed fills + boost chips + pattern signals.</div></div>
<div class="step"><div class="n">08:15</div><div class="body"><strong>Sarah drills into a $5M permit.</strong> Top candidate card: Carmen Green, Endorsed · 3 playbooks chip, boost +0.166, pattern line reads "leader archetype · 47% OSHA-10." Sarah hovers the chip — narrative tooltip: "filled Welder x2 in Toledo (2026-04-15), Welder x1 in Toledo (2026-04-18)."</div></div>
<div class="step"><div class="n">08:30</div><div class="body"><strong>Sarah calls Carmen.</strong> Clicks Call button → <code>/log</code> fires → <code>playbook_memory.seed</code> → <code>persist_sql</code> → successful_playbooks_live grows by one. Button flashes "Logged" for 1.4s. No modal, no form, no second click.</div></div>
<div class="step"><div class="n">09:00</div><div class="body"><strong>Kim (another staffer) opens the same UI.</strong> Her profile loads. Her workspaces show her own contracts. She searches "reliable forklift Chicago" — MEMORY chip shows the pattern discovered across Sarah's morning work AND prior fills. Carmen, already logged by Sarah, shows up with an updated citation count.</div></div>
<div class="step"><div class="n">12:30</div><div class="body"><strong>Client pushes 20 new contracts + 1M ATS delta.</strong> Ch7 scale flow fires. Ingest in seconds; embedding refresh kicks off as a background job. Searches continue against old embeddings.</div></div>
<div class="step"><div class="n">14:00</div><div class="body"><strong>Emergency: worker Dave no-showed.</strong> Sarah clicks No-show button on Dave's card → <code>/log_failure</code> → <code>mark_failed</code> records a penalty. Next similar query dampens Dave's boost by 0.5. Sarah continues the refill — the refill excludes Dave and the 2 others already booked for this shift.</div></div>
<div class="step"><div class="n">15:00</div><div class="body"><strong>New embeddings live.</strong> Hot-swap promotion. Searches now see all 1M new profiles. Sarah's noon query re-run would produce different top-5.</div></div>
<div class="step"><div class="n">17:00</div><div class="body"><strong>End-of-day retrospective.</strong> Any staffer who ran <code>tests/multi-agent/scenario.ts</code> gets <code>report.md</code> auto-generated. Workspace activity logs aggregate per staffer. <code>GET /vectors/playbook_memory/status</code> shows active vs retired counts. KB indexes the run (<code>kb.indexRun</code>) and the overview model synthesizes a pathway recommendation for the next matching signature. Every event outcome has already streamed to <code>lakehouse-observer.service</code> on :3800 for ERROR_ANALYZER + PLAYBOOK_BUILDER consumption.</div></div>
<div class="step"><div class="n">17:15</div><div class="body"><strong>Kim fires a natural-language query from the search box.</strong> "need 3 forklift operators in Joliet by Monday" → <code>POST /memory/query</code> on the Bun MCP. Regex normalizer extracts role / city / count / deadline / intent in 0ms (no LLM call). The unified response returns playbook workers (auto-surfaced reliable performers for Joliet Forklift with citation counts), pathway recommendation from KB, prior T3 lessons for Joliet, and top staffers by competence — all in ~300ms.</div></div>
<div class="step"><div class="n">22:00</div><div class="body"><strong>Overnight trial cycle.</strong> Autotune agent continues in the background. Trial journal grows. KB's <code>detectErrorCorrections</code> scans today's outcomes for fail→succeed deltas on the same signature; any correction gets logged to <code>data/_kb/error_corrections.jsonl</code> with the config diff. Tomorrow morning, the system is measurably better at something it got asked about today.</div></div>
<h3>SMS + email drafts in the pipeline</h3>
<p>After each sealed fill (via scenario.ts or manual <code>/log</code> flow with downstream hooks), <code>generateArtifacts</code> in the scenario runner produces: (a) one SMS per worker (TO: Name, message under 180 chars), (b) one client confirmation email. Drafts are saved to <code>sms.md</code> and <code>emails.md</code> under the scenario output dir. Ollama drafts them; the staffer reviews and sends. No auto-send; human-in-the-loop.</p>
</div>
<!-- ═══ 11. LIMITS & NON-GOALS ═══ -->
<div class="chapter" id="ch11">
<div class="num">Chapter 11</div>
<h2>Known limits &amp; non-goals</h2>
<div class="lede">Honesty is a feature. Everything below is either deferred or explicitly out of scope.</div>
<h4>Deferred — real architectural work, just not shipped yet</h4>
<ul>
<li><strong>Rate / margin awareness.</strong> Worker pay expectations vs contract bill rate not modeled. Requires adding <code>pay_rate</code> to workers, <code>bill_rate</code> to contracts, and a filter + warning path. Partially addressed via <code>ContractTerms.budget_per_hour_max</code> passed to T3/rescue prompts, but the match-time filter isn't wired yet.</li>
<li><strong>Mem0-style UPDATE / DELETE / NOOP operations on playbooks.</strong> Today <code>/seed</code> only ADDs. Same <code>(operation, date)</code> pair appends a duplicate instead of refining an existing entry. Phase 26 item — cheap to add, moderate payoff.</li>
<li><strong>Letta working-memory hot cache.</strong> Every boost query scans all active playbook entries from in-memory state. 1.9K today; cheap. Will bite somewhere north of 100K. LRU for the last-N playbooks or current-sig neighborhood deferred until that ceiling approaches.</li>
<li><strong>Chunking cache (Phase 21 Rust port).</strong> TS primitives <code>generateContinuable</code> + <code>generateTreeSplit</code> are wired, but <code>crates/aibridge/src/{continuation.rs, tree_split.rs}</code> + <code>crates/storaged/src/chunk_cache.rs</code> remain queued. Gateway-side callers currently don't have the same protection against silent truncation that the TS test harness does.</li>
<li><strong>Confidence calibration.</strong> Top-K is a rank, not a probability. No calibrated "85% likely to accept" score. Requires outcome-labeled training data.</li>
<li><strong>Neural re-ranker.</strong> Phase 19 is statistical + semantic (now with geo + role prefilter, Phase 25 retirement). A (query, candidate, outcome)-trained re-ranker is deferred only if the statistical floor plateaus below usable recall — current 14× citation lift on identical inputs suggests it hasn't.</li>
<li><strong>Observer → autotune feedback wire.</strong> Phase 24 streams scenario outcomes into <code>data/_observer/ops.jsonl</code>; autotune agent still runs on its own HNSW-trial schedule and hasn't subscribed to the outcome metric stream yet. Phase 26+ item — connects the last loop.</li>
<li><strong>call_log cross-reference.</strong> Infrastructure present; current synthetic candidates table is too small to cross-ref. Fixes when real ATS lands.</li>
</ul>
<h4>Non-goals — explicitly out of scope</h4>
<ul>
<li><strong>Cloud deployment.</strong> Local-first by design. Works offline after setup.</li>
<li><strong>Full ACID transactions.</strong> Single-writer model is sufficient; Delta Lake-grade MVCC is deliberately not attempted.</li>
<li><strong>Real-time streaming / CDC.</strong> Batch ingest is the model. Scheduled refresh, not transactional replication.</li>
<li><strong>Replacing the CRM.</strong> This is the analytical + AI layer <em>behind</em> the CRM. Operational CRUD stays with the existing system.</li>
<li><strong>Custom file formats.</strong> Parquet for datasets, sidecar indexes for vectors. No proprietary formats (ADR-008, ADR-018 reaffirm).</li>
<li><strong>Hard multi-tenant isolation.</strong> Profiles and federation provide soft isolation. Adversarial multi-tenant is not a goal — this system assumes a single-trust operator.</li>
</ul>
<div class="narr">
<strong>Overall bet.</strong> The substrate is conservative: Parquet + DataFusion + HNSW + Ollama + object storage. Every layer is replaceable, open, auditable. The intelligence layer (playbook_memory, patterns, autotune) is statistical, not neural — cheaper, explainable, rebuildable from the journal alone. If the statistical floor plateaus below what a real client needs, Phase 20+ adds neural re-rank on top. We don't make that call until measurement demands it.
</div>
</div>
</div>
</div>
<div class="footer">Lakehouse spec · v2 2026-04-21 · Phases 19-25 shipped (playbook boost, model matrix, continuation, KB, staffer competence, observer ingest, validity windows) · maintained from <code>docs/DECISIONS.md</code> · <a href="proof">architecture live-tested</a> · <a href="console">walkthrough</a></div>
</body></html>
mcp-server/workspaces.html
+487
View File
@@ -0,0 +1,487 @@
<!DOCTYPE html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>Lakehouse — Workspaces</title>
<style>
*{margin:0;padding:0;box-sizing:border-box}
body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;color:#b0b8c4;font-size:14px;line-height:1.55;-webkit-font-smoothing:antialiased}
a{color:#58a6ff;text-decoration:none}
a:hover{color:#79c0ff}
.bar{background:#0d1117;padding:0 24px;height:56px;border-bottom:1px solid #171d27;display:flex;justify-content:space-between;align-items:center;position:sticky;top:0;z-index:10}
.bar h1{font-size:14px;font-weight:600;color:#e6edf3;letter-spacing:-0.2px}
.bar nav{display:flex;gap:2px;flex-wrap:wrap}
.bar nav a{font-size:12px;color:#545d68;padding:6px 14px;border-radius:6px;transition:all 0.15s}
.bar nav a:hover{color:#e6edf3;background:#161b22}
.bar nav a.active{color:#e6edf3;background:#1c2333}
.wrap{max-width:1200px;margin:0 auto;padding:24px 20px 60px}
h2{color:#e6edf3;font-size:22px;font-weight:700;letter-spacing:-0.3px;margin-bottom:4px}
.lede{color:#8b949e;font-size:13px;margin-bottom:18px;line-height:1.6;max-width:720px}
.layout{display:grid;grid-template-columns:320px 1fr;gap:16px}
.list{display:flex;flex-direction:column;gap:8px;max-height:calc(100vh - 200px);overflow-y:auto;padding-right:4px}
.ws-card{background:#0d1117;border:1px solid #171d27;border-radius:10px;padding:12px 14px;cursor:pointer;transition:all 0.15s}
.ws-card:hover{border-color:#30363d;background:#161b22}
.ws-card.active{border-color:#1f6feb;background:#0d1d38}
.ws-card .name{color:#e6edf3;font-weight:600;font-size:13px;margin-bottom:4px;line-height:1.4}
.ws-card .meta{color:#8b949e;font-size:11px}
.ws-card .tier{display:inline-block;padding:1px 7px;border-radius:8px;font-size:9px;font-weight:600;margin-right:6px;letter-spacing:0.3px}
.tier.daily{background:#2d1b00;color:#d29922;border:1px solid #854d0e}
.tier.weekly{background:#0d2340;color:#58a6ff;border:1px solid #1f6feb80}
.tier.monthly{background:#1e1047;color:#a78bfa;border:1px solid #5b21b680}
.tier.pinned{background:#0d2818;color:#3fb950;border:1px solid #2ea04380}
.detail{background:#0d1117;border:1px solid #171d27;border-radius:12px;padding:18px;min-height:400px}
.section{margin-bottom:20px}
.section-hdr{display:flex;justify-content:space-between;align-items:center;margin-bottom:10px}
.section-hdr .title{color:#545d68;font-size:11px;text-transform:uppercase;letter-spacing:1.2px;font-weight:600}
.section-hdr .count{color:#8b949e;font-size:11px}
.entry{background:#161b22;border-radius:6px;padding:10px 12px;margin-bottom:6px;font-size:12px}
.entry .title{color:#e6edf3;font-weight:500;font-size:13px}
.entry .meta{color:#8b949e;font-size:11px;margin-top:2px}
.entry .sub{color:#545d68;font-size:11px;margin-top:2px;font-family:ui-monospace,Menlo,monospace}
.empty{color:#484f58;font-size:12px;font-style:italic;padding:10px 12px}
.btn{padding:8px 14px;background:#1f6feb;border:none;border-radius:6px;color:#fff;font-size:12px;font-weight:600;cursor:pointer}
.btn:hover{background:#388bfd}
.btn:disabled{opacity:0.4;cursor:not-allowed}
.btn.ghost{background:transparent;border:1px solid #21262d;color:#c9d1d9}
.btn.ghost:hover{background:#161b22}
.btn.green{background:#2ea043}
.btn.green:hover{background:#3fb950}
.btn.sm{padding:4px 10px;font-size:11px}
.form{display:flex;flex-direction:column;gap:10px}
.form label{color:#8b949e;font-size:11px;text-transform:uppercase;letter-spacing:1px;font-weight:600;margin-bottom:-6px}
.form input,.form textarea,.form select{padding:8px 12px;background:#161b22;border:1px solid #21262d;border-radius:6px;color:#e6edf3;font-size:13px;outline:none;font-family:inherit}
.form input:focus,.form textarea:focus,.form select:focus{border-color:#388bfd}
.form textarea{resize:vertical;min-height:60px;font-family:inherit}
.form-row{display:flex;gap:10px}
.form-row>*{flex:1}
.modal-backdrop{position:fixed;inset:0;background:rgba(0,0,0,0.7);z-index:100;display:none;align-items:center;justify-content:center;padding:20px}
.modal-backdrop.show{display:flex}
.modal{background:#0d1117;border:1px solid #30363d;border-radius:12px;padding:22px;max-width:520px;width:100%;max-height:90vh;overflow-y:auto}
.modal h3{color:#e6edf3;font-size:17px;font-weight:700;margin-bottom:12px}
.hdr-block{padding-bottom:14px;border-bottom:1px solid #171d27;margin-bottom:16px}
.hdr-block .wname{color:#e6edf3;font-size:18px;font-weight:700;margin-bottom:4px}
.hdr-block .wmeta{color:#8b949e;font-size:12px}
.hdr-block .desc{color:#c9d1d9;font-size:13px;margin-top:8px;line-height:1.6}
.prev-owners{display:flex;flex-wrap:wrap;gap:4px;margin-top:6px}
.prev-owners .chip{padding:2px 8px;border-radius:10px;font-size:10px;background:#161b22;color:#8b949e;border:1px solid #21262d}
.actions{display:flex;gap:8px;flex-wrap:wrap;margin-top:12px}
.flash{padding:10px 14px;border-radius:6px;margin-top:10px;font-size:12px}
.flash.ok{background:#0d2818;color:#86efac;border:1px solid #2ea04360}
.flash.err{background:#3a1a1a;color:#fca5a5;border:1px solid #f8514960}
.activity{max-height:240px;overflow-y:auto}
.activity .entry{border-left:2px solid #2ea043}
.footer{border-top:1px solid #171d27;padding:20px;text-align:center;color:#3d444d;font-size:11px}
@media(max-width:900px){
.layout{grid-template-columns:1fr}
.list{max-height:none}
.bar nav{font-size:10px}
}
</style></head>
<body>
<div class="bar">
<h1>Lakehouse — Workspaces</h1>
<nav>
<a href=".">Dashboard</a>
<a href="console">Walkthrough</a>
<a href="proof">Architecture</a>
<a href="spec">Spec</a>
<a href="onboard">Onboard</a>
<a href="alerts">Alerts</a>
<a href="workspaces" class="active">Workspaces</a>
</nav>
</div>
<div class="wrap">
<div style="margin-bottom:18px">
<h2>Per-contract workspaces</h2>
<div class="lede">
Each contract gets its own workspace: saved searches, shortlisted candidates, activity log, tier (daily / weekly / monthly / pinned). Persisted across sessions. One-click zero-copy handoff between staffers — Sarah hands off to Kim, Kim opens the workspace and sees everything Sarah saw. This is the layer that keeps institutional memory per-contract, not per-person.
</div>
<button class="btn" id="new-ws-btn">New workspace</button>
</div>
<div class="layout">
<div class="list" id="ws-list"><div class="empty">Loading workspaces…</div></div>
<div class="detail" id="ws-detail">
<div class="empty">Select a workspace from the list, or create one.</div>
</div>
</div>
</div>
<!-- Modals -->
<div class="modal-backdrop" id="modal-new">
<div class="modal">
<h3>New workspace</h3>
<form class="form" id="form-new" onsubmit="return false">
<label>Contract / fill name</label>
<input type="text" id="nw-name" placeholder="e.g. Riverfront Steel — Week 17 fills" required>
<label>Description</label>
<textarea id="nw-desc" placeholder="What's this contract about?"></textarea>
<div class="form-row">
<div>
<label>Tier</label>
<select id="nw-tier">
<option value="daily">Daily</option>
<option value="weekly" selected>Weekly</option>
<option value="monthly">Monthly</option>
<option value="pinned">Pinned</option>
</select>
</div>
<div>
<label>Owner (staffer)</label>
<input type="text" id="nw-owner" placeholder="sarah" value="sarah">
</div>
</div>
<div class="actions">
<button type="button" class="btn green" id="nw-create">Create</button>
<button type="button" class="btn ghost" data-close="modal-new">Cancel</button>
</div>
<div id="nw-result"></div>
</form>
</div>
</div>
<div class="modal-backdrop" id="modal-handoff">
<div class="modal">
<h3>Hand off to another staffer</h3>
<div style="color:#8b949e;font-size:12px;margin-bottom:14px;line-height:1.6">
Transfers ownership of this workspace. The prior owner is preserved in <code>previous_owners</code> for the audit trail.
The recipient sees all saved searches, shortlist entries, and activity from the prior owner — no copy, no migration.
</div>
<form class="form" onsubmit="return false">
<label>New owner</label>
<input type="text" id="ho-owner" placeholder="kim">
<label>Reason</label>
<input type="text" id="ho-reason" placeholder="e.g. end of shift, taking over for Sarah">
<div class="actions">
<button type="button" class="btn green" id="ho-submit">Hand off</button>
<button type="button" class="btn ghost" data-close="modal-handoff">Cancel</button>
</div>
<div id="ho-result"></div>
</form>
</div>
</div>
<div class="modal-backdrop" id="modal-shortlist">
<div class="modal">
<h3>Shortlist a candidate</h3>
<form class="form" onsubmit="return false">
<label>Dataset</label>
<input type="text" id="sl-dataset" placeholder="workers_500k" value="workers_500k">
<label>Worker ID</label>
<input type="text" id="sl-record" placeholder="e.g. 4661 or W500K-4661">
<label>Notes</label>
<textarea id="sl-notes" placeholder="Why this candidate — skills, past fills, availability window"></textarea>
<div class="actions">
<button type="button" class="btn green" id="sl-submit">Add to shortlist</button>
<button type="button" class="btn ghost" data-close="modal-shortlist">Cancel</button>
</div>
<div id="sl-result"></div>
</form>
</div>
</div>
<div class="modal-backdrop" id="modal-search">
<div class="modal">
<h3>Save a search</h3>
<form class="form" onsubmit="return false">
<label>Search name</label>
<input type="text" id="sv-name" placeholder="e.g. Available Toledo welders with OSHA">
<label>SQL (will run against the catalog via /query/sql)</label>
<textarea id="sv-sql" placeholder="SELECT * FROM workers_500k WHERE role='Welder' AND city='Toledo' AND state='OH' AND CAST(availability AS DOUBLE) > 0.5 LIMIT 20"></textarea>
<div class="actions">
<button type="button" class="btn green" id="sv-submit">Save search</button>
<button type="button" class="btn ghost" data-close="modal-search">Cancel</button>
</div>
<div id="sv-result"></div>
</form>
</div>
</div>
<div class="modal-backdrop" id="modal-activity">
<div class="modal">
<h3>Log activity</h3>
<form class="form" onsubmit="return false">
<label>Action</label>
<input type="text" id="act-action" placeholder="e.g. called, emailed, shortlisted, rejected">
<label>Detail</label>
<textarea id="act-detail" placeholder="What happened and what's the next step"></textarea>
<div class="actions">
<button type="button" class="btn green" id="act-submit">Log activity</button>
<button type="button" class="btn ghost" data-close="modal-activity">Cancel</button>
</div>
<div id="act-result"></div>
</form>
</div>
</div>
<div class="footer">Lakehouse · workspaces · per-contract state, zero-copy handoff (Phase 8.5)</div>
<script>
var P=location.pathname.indexOf('/lakehouse')>=0?'/lakehouse':'';
var A=location.origin+P;
var state={list:[], current:null, currentAgent:'sarah'};
function el(t,c,x){var e=document.createElement(t);if(c)e.className=c;if(x!==undefined&&x!==null)e.textContent=String(x);return e}
function apiGet(p){
return fetch(A+'/api'+p).then(function(r){return r.ok?r.json():r.text().then(function(b){throw new Error('HTTP '+r.status+': '+b)})});
}
function apiPost(p,body){
return fetch(A+'/api'+p,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(body||{})}).then(function(r){
return r.text().then(function(t){
if(!r.ok) throw new Error('HTTP '+r.status+': '+t.slice(0,200));
try{return JSON.parse(t)}catch(e){return {raw:t}}
});
});
}
function showModal(id){ document.getElementById(id).classList.add('show') }
function hideModal(id){ document.getElementById(id).classList.remove('show') }
function loadList(preselectId){
apiGet('/workspaces').then(function(arr){
if(!Array.isArray(arr)) arr=[];
state.list=arr;
renderList();
if(preselectId){ openWorkspace(preselectId); }
else if(arr.length>0 && !state.current){ openWorkspace(arr[0].id); }
}).catch(function(e){
var host=document.getElementById('ws-list');host.textContent='';
host.appendChild(el('div','empty','Failed to load: '+e.message));
});
}
function renderList(){
var host=document.getElementById('ws-list');host.textContent='';
if(state.list.length===0){
host.appendChild(el('div','empty','No workspaces yet. Create one to get started.'));
return;
}
state.list.slice().sort(function(a,b){
return (b.updated_at||'').localeCompare(a.updated_at||'');
}).forEach(function(w){
var card=el('div','ws-card'+(state.current&&state.current.id===w.id?' active':''));
card.onclick=function(){ openWorkspace(w.id); };
card.appendChild(el('div','name',w.name||'(unnamed)'));
var meta=el('div','meta');
meta.appendChild(el('span','tier '+(w.tier||'weekly'),(w.tier||'weekly').toUpperCase()));
meta.appendChild(document.createTextNode((w.owner||'?')+' · '+((w.shortlist||[]).length)+' shortlisted · '+((w.activity||[]).length)+' events'));
card.appendChild(meta);
host.appendChild(card);
});
}
function openWorkspace(id){
apiGet('/workspaces/'+encodeURIComponent(id)).then(function(w){
if(!w||w.error){ throw new Error(w&&w.error?w.error:'not found'); }
state.current=w;
renderList();
renderDetail();
}).catch(function(e){
var host=document.getElementById('ws-detail');host.textContent='';
host.appendChild(el('div','empty','Failed to load: '+e.message));
});
}
function renderDetail(){
var host=document.getElementById('ws-detail');host.textContent='';
var w=state.current;
if(!w){ host.appendChild(el('div','empty','Select a workspace from the list, or create one.')); return; }
// Header
var hdr=el('div','hdr-block');
hdr.appendChild(el('div','wname',w.name||'(unnamed)'));
var meta=el('div','wmeta');
meta.appendChild(el('span','tier '+(w.tier||'weekly'),(w.tier||'weekly').toUpperCase()));
meta.appendChild(document.createTextNode(' owner: '+(w.owner||'?')+' · created '+(w.created_at||'').slice(0,10)+' · updated '+(w.updated_at||'').slice(0,10)));
hdr.appendChild(meta);
if(w.description){ hdr.appendChild(el('div','desc',w.description)); }
if((w.previous_owners||[]).length>0){
var pos=el('div','prev-owners');
pos.appendChild(el('span','chip','Previous owners:'));
w.previous_owners.forEach(function(p){ pos.appendChild(el('span','chip',p)); });
hdr.appendChild(pos);
}
// Action buttons
var actions=el('div','actions');
var handoffBtn=el('button','btn','Hand off');
handoffBtn.onclick=function(){ showModal('modal-handoff'); };
actions.appendChild(handoffBtn);
var slBtn=el('button','btn ghost','Shortlist candidate');
slBtn.onclick=function(){ showModal('modal-shortlist'); };
actions.appendChild(slBtn);
var svBtn=el('button','btn ghost','Save search');
svBtn.onclick=function(){ showModal('modal-search'); };
actions.appendChild(svBtn);
var actBtn=el('button','btn ghost','Log activity');
actBtn.onclick=function(){ showModal('modal-activity'); };
actions.appendChild(actBtn);
hdr.appendChild(actions);
host.appendChild(hdr);
// Shortlist
host.appendChild(sectionHeader('Shortlist',(w.shortlist||[]).length+' candidates'));
if((w.shortlist||[]).length===0){
host.appendChild(el('div','empty','No candidates shortlisted yet.'));
} else {
w.shortlist.forEach(function(sl){
var e=el('div','entry');
e.appendChild(el('div','title',(sl.dataset||'?')+' · '+(sl.record_id||'?')));
if(sl.notes) e.appendChild(el('div','meta',sl.notes));
e.appendChild(el('div','sub','by '+(sl.by_agent||sl.agent||'?')+' · '+(sl.at||sl.added_at||'').slice(0,16).replace('T',' ')));
host.appendChild(e);
});
}
// Saved searches
host.appendChild(sectionHeader('Saved searches',(w.saved_searches||[]).length+' queries'));
if((w.saved_searches||[]).length===0){
host.appendChild(el('div','empty','No saved searches yet.'));
} else {
w.saved_searches.forEach(function(sv){
var e=el('div','entry');
e.appendChild(el('div','title',sv.name||'(unnamed)'));
if(sv.sql) e.appendChild(el('div','sub',sv.sql));
e.appendChild(el('div','sub','by '+(sv.by_agent||sv.agent||'?')+' · '+(sv.at||sv.added_at||'').slice(0,16).replace('T',' ')));
host.appendChild(e);
});
}
// Activity
host.appendChild(sectionHeader('Activity',(w.activity||[]).length+' events'));
if((w.activity||[]).length===0){
host.appendChild(el('div','empty','No activity logged yet.'));
} else {
var act=el('div','activity');
w.activity.slice().reverse().forEach(function(a){
var e=el('div','entry');
e.appendChild(el('div','title',a.action||'(no action)'));
if(a.detail) e.appendChild(el('div','meta',a.detail));
e.appendChild(el('div','sub','by '+(a.by_agent||a.agent||'?')+' · '+(a.at||'').slice(0,16).replace('T',' ')));
act.appendChild(e);
});
host.appendChild(act);
}
}
function sectionHeader(title,count){
var s=el('div','section-hdr');
s.appendChild(el('span','title',title));
s.appendChild(el('span','count',count));
return s;
}
function flash(id,text,kind){
var box=document.getElementById(id);
box.textContent='';
box.appendChild(el('div','flash '+(kind||'ok'),text));
}
function createWorkspace(){
var name=document.getElementById('nw-name').value.trim();
var desc=document.getElementById('nw-desc').value.trim();
var tier=document.getElementById('nw-tier').value;
var owner=document.getElementById('nw-owner').value.trim()||'sarah';
if(!name){ flash('nw-result','Name is required','err');return; }
apiPost('/workspaces/create',{name:name,description:desc,owner:owner,tier:tier}).then(function(r){
state.currentAgent=owner;
hideModal('modal-new');
document.getElementById('nw-name').value='';
document.getElementById('nw-desc').value='';
loadList(r.id);
}).catch(function(e){ flash('nw-result',e.message,'err'); });
}
function submitHandoff(){
var w=state.current;if(!w)return;
var to=document.getElementById('ho-owner').value.trim();
var reason=document.getElementById('ho-reason').value.trim();
if(!to){ flash('ho-result','New owner required','err');return; }
apiPost('/workspaces/'+encodeURIComponent(w.id)+'/handoff',{to_agent:to,reason:reason}).then(function(){
state.currentAgent=to;
hideModal('modal-handoff');
loadList(w.id);
}).catch(function(e){ flash('ho-result',e.message,'err'); });
}
function submitShortlist(){
var w=state.current;if(!w)return;
var dataset=document.getElementById('sl-dataset').value.trim()||'workers_500k';
var record=document.getElementById('sl-record').value.trim();
var notes=document.getElementById('sl-notes').value.trim();
if(!record){ flash('sl-result','Worker ID required','err');return; }
apiPost('/workspaces/'+encodeURIComponent(w.id)+'/shortlist',{dataset:dataset,record_id:record,notes:notes,agent:state.currentAgent}).then(function(){
hideModal('modal-shortlist');
document.getElementById('sl-record').value='';
document.getElementById('sl-notes').value='';
loadList(w.id);
}).catch(function(e){ flash('sl-result',e.message,'err'); });
}
function submitSaveSearch(){
var w=state.current;if(!w)return;
var name=document.getElementById('sv-name').value.trim();
var sql=document.getElementById('sv-sql').value.trim();
if(!name||!sql){ flash('sv-result','Name and SQL are required','err');return; }
apiPost('/workspaces/'+encodeURIComponent(w.id)+'/search',{name:name,sql:sql,agent:state.currentAgent}).then(function(){
hideModal('modal-search');
document.getElementById('sv-name').value='';
document.getElementById('sv-sql').value='';
loadList(w.id);
}).catch(function(e){ flash('sv-result',e.message,'err'); });
}
function submitActivity(){
var w=state.current;if(!w)return;
var action=document.getElementById('act-action').value.trim();
var detail=document.getElementById('act-detail').value.trim();
if(!action){ flash('act-result','Action required','err');return; }
apiPost('/workspaces/'+encodeURIComponent(w.id)+'/activity',{action:action,detail:detail,agent:state.currentAgent}).then(function(){
hideModal('modal-activity');
document.getElementById('act-action').value='';
document.getElementById('act-detail').value='';
loadList(w.id);
}).catch(function(e){ flash('act-result',e.message,'err'); });
}
document.addEventListener('DOMContentLoaded',function(){
document.getElementById('new-ws-btn').addEventListener('click',function(){ showModal('modal-new'); });
document.getElementById('nw-create').addEventListener('click',createWorkspace);
document.getElementById('ho-submit').addEventListener('click',submitHandoff);
document.getElementById('sl-submit').addEventListener('click',submitShortlist);
document.getElementById('sv-submit').addEventListener('click',submitSaveSearch);
document.getElementById('act-submit').addEventListener('click',submitActivity);
Array.prototype.forEach.call(document.querySelectorAll('[data-close]'),function(b){
b.addEventListener('click',function(){ hideModal(b.getAttribute('data-close')); });
});
// Click-outside-to-close
Array.prototype.forEach.call(document.querySelectorAll('.modal-backdrop'),function(m){
m.addEventListener('click',function(e){ if(e.target===m) m.classList.remove('show'); });
});
loadList();
});
</script>
</body></html>
scripts/ab_t3_test.sh
Executable
+77
View File
@@ -0,0 +1,77 @@
#!/usr/bin/env bash
# A/B test of T3 overseer: does it actually make subsequent runs better?
# Chains Run B (T3 seed) → Run C (T3 + read-back) → Run D (T3 cloud).
# Run A is assumed already complete (launched separately). Aggregates
# metrics at the end into ab_scorecard.json.
set -e
cd "$(dirname "$0")/.."
export OLLAMA_CLOUD_KEY="$(python3 -c "import json; print(json.load(open('/root/llm_team_config.json'))['providers']['ollama_cloud']['api_key'])")"
echo "▶ A/B test start at $(date -Iseconds)"
echo "▶ prior lessons dir: $(ls data/_playbook_lessons 2>/dev/null | wc -l) files"
# Run B — T3 enabled local, no prior lessons should exist yet
echo "──── RUN B: T3 local, seeds first lesson ────"
bun tests/multi-agent/scenario.ts > /tmp/lakehouse_ab_B.log 2>&1 || true
echo " B exit=$?"
ls data/_playbook_lessons/*.json 2>/dev/null | head -5
# Run C — T3 enabled local, B's lesson should load
echo "──── RUN C: T3 local, reads B's lesson ────"
bun tests/multi-agent/scenario.ts > /tmp/lakehouse_ab_C.log 2>&1 || true
echo " C exit=$?"
# Run D — T3 enabled CLOUD (gpt-oss:120b), reads B+C lessons
echo "──── RUN D: T3 cloud, reads B+C lessons ────"
LH_OVERVIEW_CLOUD=1 bun tests/multi-agent/scenario.ts > /tmp/lakehouse_ab_D.log 2>&1 || true
echo " D exit=$?"
echo "▶ all runs done at $(date -Iseconds)"
echo "▶ scorecard:"
ls -1dt tests/multi-agent/playbooks/scenario-* | head -4 | tac | python3 -c "
import sys, os, json
runs = [l.strip() for l in sys.stdin if l.strip()]
labels = ['A(no-T3)','B(T3-seed)','C(T3-read)','D(T3-cloud)']
# Prepend Run A: most recent BEFORE the ab_t3_test kicked off is Run A
# (launched separately). But we only picked up the most recent 4 runs.
# Actually: ab_t3_test runs B/C/D, so recent 3 = B,C,D. Run A is the one
# BEFORE those — find it separately.
# Reread to include Run A:
import subprocess
all_runs = subprocess.check_output(['bash','-c','ls -1dt tests/multi-agent/playbooks/scenario-* | head -8']).decode().strip().split('\n')
# The 4 most recent are D, C, B, A (reverse chronological).
top4 = list(reversed(all_runs[:4])) # oldest first → A,B,C,D
rows = []
for i, path in enumerate(top4):
try:
results = json.load(open(os.path.join(path, 'results.json')))
except FileNotFoundError:
continue
ok = sum(1 for r in results if r.get('ok'))
turns = sum(r.get('turns', 0) for r in results)
gaps = sum(len(r.get('gap_signals', [])) for r in results)
cites = sum(len(r.get('playbook_citations') or []) for r in results)
prior = []
try:
prior = json.load(open(os.path.join(path, 'prior_lessons.json')))
except FileNotFoundError:
pass
rows.append({
'label': labels[i] if i < len(labels) else f'run{i}',
'path': path,
'ok_events': ok,
'total_events': len(results),
'total_turns': turns,
'total_gaps': gaps,
'total_citations': cites,
'prior_lessons_loaded': len(prior),
})
scorecard = {'generated_at': __import__('datetime').datetime.utcnow().isoformat()+'Z', 'runs': rows}
open('tests/multi-agent/playbooks/ab_scorecard.json','w').write(json.dumps(scorecard, indent=2))
print(json.dumps(scorecard, indent=2))
"
echo "▶ saved: tests/multi-agent/playbooks/ab_scorecard.json"
scripts/kb_measure.py
Executable
+163
View File
@@ -0,0 +1,163 @@
#!/usr/bin/env python3
"""Aggregate KB state for item 3 decision.
Reads data/_kb/*.jsonl and tests/multi-agent/playbooks/*/results.json
to answer:
- How many distinct signatures exist?
- Total runs, avg ok rate, avg citations per event?
- Which (role, city) combos have NEVER gotten a citation?
- Recommender confidence progression (cold → medium → high)?
- Mean turn count trend across runs (proxy for efficiency).
Run after `scripts/run_kb_batch.sh` completes. Writes a markdown
summary to tests/multi-agent/playbooks/kb_measurement.md and prints
to stdout.
"""
import json
import os
import sys
from collections import Counter, defaultdict
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
KB = ROOT / "data" / "_kb"
PLAYBOOKS = ROOT / "tests" / "multi-agent" / "playbooks"
def load_jsonl(p):
if not p.exists():
return []
out = []
for line in p.read_text().splitlines():
if line.strip():
try:
out.append(json.loads(line))
except json.JSONDecodeError:
pass
return out
def main():
sigs = load_jsonl(KB / "signatures.jsonl")
outcomes = load_jsonl(KB / "outcomes.jsonl")
recs = load_jsonl(KB / "pathway_recommendations.jsonl")
corrections = load_jsonl(KB / "error_corrections.jsonl")
# --- Basic counts ---
print(f"Signatures: {len(sigs)}")
print(f"Outcomes: {len(outcomes)}")
print(f"Recommendations: {len(recs)}")
print(f"Error corrections: {len(corrections)}")
print()
# --- Recommender confidence progression ---
conf_counts = Counter(r.get("confidence", "?") for r in recs)
print(f"Recommender confidence distribution:")
for c in ("high", "medium", "low"):
print(f" {c:8s}: {conf_counts.get(c, 0)}")
print()
# Time-ordered confidence
recs_sorted = sorted(recs, key=lambda r: r.get("generated_at", ""))
neighbor_counts = [len(r.get("neighbors_consulted", [])) for r in recs_sorted]
if neighbor_counts:
print(f"Neighbors consulted over time (first → last):")
print(f" first 3: {neighbor_counts[:3]}")
print(f" last 3: {neighbor_counts[-3:]}")
print(f" max: {max(neighbor_counts)}")
print()
# --- Fill rate + citation density per run ---
if outcomes:
total_ok = sum(o["ok_events"] for o in outcomes)
total_events = sum(o["total_events"] for o in outcomes)
total_cites = sum(o.get("total_citations", 0) for o in outcomes)
total_turns = sum(o.get("total_turns", 0) for o in outcomes)
print(f"Fill rate: {total_ok}/{total_events} = {100*total_ok/max(1,total_events):.1f}%")
print(f"Avg citations per run: {total_cites/len(outcomes):.2f}")
print(f"Avg turns per run: {total_turns/len(outcomes):.1f}")
print()
# First 5 runs vs last 5 — does it get better?
sorted_out = sorted(outcomes, key=lambda o: o.get("created_at", ""))
if len(sorted_out) >= 10:
first = sorted_out[:5]
last = sorted_out[-5:]
fok = sum(o["ok_events"] for o in first) / sum(o["total_events"] for o in first)
lok = sum(o["ok_events"] for o in last) / sum(o["total_events"] for o in last)
fcit = sum(o.get("total_citations", 0) for o in first) / 5
lcit = sum(o.get("total_citations", 0) for o in last) / 5
print(f"First 5 runs ok rate: {100*fok:.1f}% avg cites: {fcit:.2f}")
print(f"Last 5 runs ok rate: {100*lok:.1f}% avg cites: {lcit:.2f}")
print()
# --- Per-(role, city) citation coverage ---
cite_by_combo = Counter()
combo_attempts = Counter()
for o in outcomes:
for ev in o.get("per_event", []):
key = (ev.get("role", "?"), "?") # city not in per_event summary
combo_attempts[key] += 1
# Read the playbook dirs for full event detail (has city)
cites_by_role_city = defaultdict(lambda: {"attempts": 0, "citations": 0, "ok": 0})
for o in outcomes:
run_dir = PLAYBOOKS / o["run_id"]
results_file = run_dir / "results.json"
if not results_file.exists():
continue
try:
results = json.loads(results_file.read_text())
except Exception:
continue
for r in results:
e = r.get("event", {})
key = (e.get("role"), e.get("city"), e.get("state"))
cites_by_role_city[key]["attempts"] += 1
cites_by_role_city[key]["citations"] += len(r.get("playbook_citations") or [])
if r.get("ok"):
cites_by_role_city[key]["ok"] += 1
combos_with_cites = [(k, v) for k, v in cites_by_role_city.items() if v["citations"] > 0]
combos_zero_cites = [(k, v) for k, v in cites_by_role_city.items() if v["citations"] == 0 and v["ok"] > 0]
print(f"(role, city, state) combos with any citation: {len(combos_with_cites)}")
print(f"(role, city, state) combos with ok fills but 0 cites: {len(combos_zero_cites)}")
print()
if combos_with_cites:
print("Top 10 combos by citation count:")
for (role, city, state), v in sorted(combos_with_cites, key=lambda x: -x[1]["citations"])[:10]:
print(f" {role:25s} {city:15s} {state}: {v['citations']} cites across {v['attempts']} attempts ({v['ok']} ok)")
print()
# --- Write markdown report ---
lines = ["# KB Measurement Report", ""]
lines.append(f"Generated from {len(outcomes)} runs across {len(sigs)} distinct signatures.")
lines.append("")
lines.append("## Recommender confidence")
for c in ("high", "medium", "low"):
lines.append(f"- {c}: {conf_counts.get(c, 0)}")
lines.append("")
lines.append("## Overall fill + citation")
if outcomes:
lines.append(f"- Fill rate: **{total_ok}/{total_events}** ({100*total_ok/max(1,total_events):.1f}%)")
lines.append(f"- Avg citations per run: **{total_cites/len(outcomes):.2f}**")
lines.append(f"- Avg turns per run: {total_turns/len(outcomes):.1f}")
lines.append("")
lines.append("## Citation coverage by (role, city, state)")
lines.append(f"- Combos with ≥1 citation: {len(combos_with_cites)}")
lines.append(f"- Combos with ok fills but 0 citations: {len(combos_zero_cites)}")
lines.append("")
lines.append("## Item 3 decision signal")
if combos_zero_cites:
lines.append("Non-zero: there are **combos that succeeded but never triggered playbook_memory boost**. Candidates for item 3 investigation:")
for (role, city, state), v in combos_zero_cites[:5]:
lines.append(f"- {role} in {city}, {state}: {v['ok']}/{v['attempts']} ok, 0 cites")
else:
lines.append("All ok combos got at least some citation firing. Boost mechanism is healthy; raising the cap may help but isn't forced.")
lines.append("")
out = PLAYBOOKS / "kb_measurement.md"
out.write_text("\n".join(lines))
print(f"✓ markdown report → {out}")
if __name__ == "__main__":
main()

Some files were not shown because too many files have changed in this diff Show More