checkpoint: matrix-agent-validated (2026-04-25)
Architectural snapshot of the lakehouse codebase at the point where the
full matrix-driven agent loop with Mem0 versioning + deletion was
validated end-to-end.
WHAT THIS REPO IS
A clean single-commit snapshot of the lakehouse code. Heavy test data
(.parquet datasets, vector indexes) excluded — see REPLICATION.md for
regen path. Full lakehouse history at git.agentview.dev/profit/lakehouse.
WHAT WAS PROVEN
- Vector retrieval across multi-corpora matrix (chicago_permits + entity
briefs + sec_tickers + distilled procedural + llm_team runs)
- Observer hand-review (cloud + heuristic fallback) gating each candidate
- Local-model agent loop (qwen3.5:latest) with tool use + scratchpad
- Playbook seal on success → next-iter retrieval surfaces it as preamble
- Mem0 versioning + deletion in pathway_memory:
* UPSERT: ADD on new workflow, UPDATE bumps replay_count on identical
* REVISE: chains versions, parent.superseded_at + superseded_by stamped
* RETIRE: marks specific trace retired with reason, excluded from retrieval
* HISTORY: walks chain root→tip, cycle-safe
KEY DIRECTORIES
- crates/vectord/src/pathway_memory.rs — Mem0 ops live here
- crates/vectord/src/playbook_memory.rs — original Mem0 reference
- tests/agent_test/ — local-model agent harness + PRD + session archives
- scripts/dump_raw_corpus.sh — MinIO bucket dump (raw test corpus)
- scripts/vectorize_raw_corpus.ts — corpus → vector indexes
- scripts/analyze_chicago_contracts.ts — real inference pipeline
- scripts/seal_agent_playbook.ts — Mem0 upsert from agent traces
Replication: see REPLICATION.md for Debian 13 clean install + cloud-only
adaptation (no local Ollama).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
profit
@@ -0,0 +1,34 @@
|
||||
# dependencies (bun install)
|
||||
node_modules
|
||||
|
||||
# output
|
||||
out
|
||||
dist
|
||||
*.tgz
|
||||
|
||||
# code coverage
|
||||
coverage
|
||||
*.lcov
|
||||
|
||||
# logs
|
||||
logs
|
||||
_.log
|
||||
report.[0-9]_.[0-9]_.[0-9]_.[0-9]_.json
|
||||
|
||||
# dotenv environment variable files
|
||||
.env
|
||||
.env.development.local
|
||||
.env.test.local
|
||||
.env.production.local
|
||||
.env.local
|
||||
|
||||
# caches
|
||||
.eslintcache
|
||||
.cache
|
||||
*.tsbuildinfo
|
||||
|
||||
# IntelliJ based IDEs
|
||||
.idea
|
||||
|
||||
# Finder (MacOS) folder config
|
||||
.DS_Store
|
||||
@@ -0,0 +1,110 @@
|
||||
# Lakehouse Agent Instructions
|
||||
|
||||
You are connected to a staffing intelligence system. Your job is to answer staffing questions, match workers to contracts, and track what works.
|
||||
|
||||
## Gateway
|
||||
|
||||
All tools are at `http://localhost:3700`. POST JSON, get JSON back.
|
||||
|
||||
## Tools
|
||||
|
||||
### `/search` — Hybrid SQL+Vector Search (use this most)
|
||||
Find workers matching structured criteria + semantic meaning.
|
||||
```json
|
||||
POST /search
|
||||
{
|
||||
"question": "reliable forklift operators with hazmat certification",
|
||||
"sql_filter": "role = 'Forklift Operator' AND state = 'IL' AND reliability > 0.8",
|
||||
"top_k": 5
|
||||
}
|
||||
```
|
||||
Every result is SQL-verified against the database. Trust the `sources` array — those workers exist with those exact skills and scores.
|
||||
|
||||
### `/sql` — Direct SQL Query
|
||||
For exact counts, aggregations, and structured lookups.
|
||||
```json
|
||||
POST /sql
|
||||
{ "sql": "SELECT role, COUNT(*) cnt FROM ethereal_workers GROUP BY role ORDER BY cnt DESC LIMIT 10" }
|
||||
```
|
||||
Tables: `ethereal_workers` (10K), `workers_100k` (100K), `candidates` (100K), `timesheets` (1M), `placements` (50K), `call_log` (800K), `email_log` (500K), `job_orders` (15K), `clients` (2K).
|
||||
|
||||
### `/match` — Match Workers to a Contract
|
||||
```json
|
||||
POST /match
|
||||
{
|
||||
"role": "Machine Operator",
|
||||
"state": "IN",
|
||||
"min_reliability": 0.8,
|
||||
"required_certs": ["OSHA-10"],
|
||||
"headcount": 5
|
||||
}
|
||||
```
|
||||
Returns qualified, SQL-verified workers ranked by semantic fit.
|
||||
|
||||
### `/worker/:id` — Get One Worker
|
||||
```
|
||||
GET /worker/4925
|
||||
```
|
||||
Returns all fields: name, role, city, skills, certifications, scores, communications.
|
||||
|
||||
### `/ask` — RAG Question
|
||||
For open-ended questions. Embeds your question, searches vector index, generates answer.
|
||||
```json
|
||||
POST /ask
|
||||
{ "question": "What kinds of workers do we have in Ohio?" }
|
||||
```
|
||||
|
||||
### `/log` — Record What Worked
|
||||
After a successful operation, log it. Future runs can query past successes.
|
||||
```json
|
||||
POST /log
|
||||
{
|
||||
"operation": "Filled 3 forklift positions in Chicago",
|
||||
"approach": "hybrid search: sql_filter role+state+reliability, vector rank by skills",
|
||||
"result": "3/3 filled, all verified, client satisfied"
|
||||
}
|
||||
```
|
||||
|
||||
### `/playbooks` — Learn From Past Success
|
||||
Before starting a task, check what worked before.
|
||||
```
|
||||
GET /playbooks?keyword=forklift
|
||||
```
|
||||
|
||||
### `/profile/:id` — Swap Model Profile
|
||||
Switch which Ollama model + data context is active.
|
||||
```
|
||||
POST /profile/agent-parquet (HNSW backend, qwen2.5)
|
||||
POST /profile/agent-lance (Lance IVF_PQ backend, mistral)
|
||||
```
|
||||
|
||||
### `/vram` — GPU Status
|
||||
```
|
||||
GET /vram
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
1. **Never hallucinate.** Only state facts that appear in tool responses. If the data doesn't support an answer, say so.
|
||||
2. **SQL first for structured questions.** "How many X in Y?" → use `/sql`. Don't guess counts.
|
||||
3. **Hybrid for matching.** When finding workers for a contract, use `/search` or `/match` with `sql_filter` so results are verified.
|
||||
4. **Log success.** After completing a task successfully, call `/log` so future agents can learn from it.
|
||||
5. **Check playbooks.** Before a complex task, call `/playbooks` to see if a similar task has been done before.
|
||||
6. **Verify before communicating.** Before drafting a message to a worker, confirm their details via `/worker/:id`.
|
||||
|
||||
## Workflow for Contract Filling
|
||||
|
||||
1. `GET /playbooks?keyword={role}` — check if this type was filled before
|
||||
2. `POST /match` with role, state, min_reliability, required_certs
|
||||
3. For each match: `GET /worker/:id` to confirm details
|
||||
4. Draft communication using confirmed worker details
|
||||
5. `POST /log` with outcome
|
||||
|
||||
## Available Profiles
|
||||
|
||||
| Profile | Backend | Model | Best for |
|
||||
|---|---|---|---|
|
||||
| `agent-parquet` | HNSW (RAM) | qwen2.5 | Fast precise search, <100K vectors |
|
||||
| `agent-lance` | IVF_PQ (disk) | mistral | Large scale, append-heavy, random access |
|
||||
|
||||
Swap when you need different capabilities. Check `/vram` before swapping.
|
||||
@@ -0,0 +1,106 @@
|
||||
|
||||
Default to using Bun instead of Node.js.
|
||||
|
||||
- Use `bun <file>` instead of `node <file>` or `ts-node <file>`
|
||||
- Use `bun test` instead of `jest` or `vitest`
|
||||
- Use `bun build <file.html|file.ts|file.css>` instead of `webpack` or `esbuild`
|
||||
- Use `bun install` instead of `npm install` or `yarn install` or `pnpm install`
|
||||
- Use `bun run <script>` instead of `npm run <script>` or `yarn run <script>` or `pnpm run <script>`
|
||||
- Use `bunx <package> <command>` instead of `npx <package> <command>`
|
||||
- Bun automatically loads .env, so don't use dotenv.
|
||||
|
||||
## APIs
|
||||
|
||||
- `Bun.serve()` supports WebSockets, HTTPS, and routes. Don't use `express`.
|
||||
- `bun:sqlite` for SQLite. Don't use `better-sqlite3`.
|
||||
- `Bun.redis` for Redis. Don't use `ioredis`.
|
||||
- `Bun.sql` for Postgres. Don't use `pg` or `postgres.js`.
|
||||
- `WebSocket` is built-in. Don't use `ws`.
|
||||
- Prefer `Bun.file` over `node:fs`'s readFile/writeFile
|
||||
- Bun.$`ls` instead of execa.
|
||||
|
||||
## Testing
|
||||
|
||||
Use `bun test` to run tests.
|
||||
|
||||
```ts#index.test.ts
|
||||
import { test, expect } from "bun:test";
|
||||
|
||||
test("hello world", () => {
|
||||
expect(1).toBe(1);
|
||||
});
|
||||
```
|
||||
|
||||
## Frontend
|
||||
|
||||
Use HTML imports with `Bun.serve()`. Don't use `vite`. HTML imports fully support React, CSS, Tailwind.
|
||||
|
||||
Server:
|
||||
|
||||
```ts#index.ts
|
||||
import index from "./index.html"
|
||||
|
||||
Bun.serve({
|
||||
routes: {
|
||||
"/": index,
|
||||
"/api/users/:id": {
|
||||
GET: (req) => {
|
||||
return new Response(JSON.stringify({ id: req.params.id }));
|
||||
},
|
||||
},
|
||||
},
|
||||
// optional websocket support
|
||||
websocket: {
|
||||
open: (ws) => {
|
||||
ws.send("Hello, world!");
|
||||
},
|
||||
message: (ws, message) => {
|
||||
ws.send(message);
|
||||
},
|
||||
close: (ws) => {
|
||||
// handle close
|
||||
}
|
||||
},
|
||||
development: {
|
||||
hmr: true,
|
||||
console: true,
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
HTML files can import .tsx, .jsx or .js files directly and Bun's bundler will transpile & bundle automatically. `<link>` tags can point to stylesheets and Bun's CSS bundler will bundle.
|
||||
|
||||
```html#index.html
|
||||
<html>
|
||||
<body>
|
||||
<h1>Hello, world!</h1>
|
||||
<script type="module" src="./frontend.tsx"></script>
|
||||
</body>
|
||||
</html>
|
||||
```
|
||||
|
||||
With the following `frontend.tsx`:
|
||||
|
||||
```tsx#frontend.tsx
|
||||
import React from "react";
|
||||
import { createRoot } from "react-dom/client";
|
||||
|
||||
// import .css files directly and it works
|
||||
import './index.css';
|
||||
|
||||
const root = createRoot(document.body);
|
||||
|
||||
export default function Frontend() {
|
||||
return <h1>Hello, world!</h1>;
|
||||
}
|
||||
|
||||
root.render(<Frontend />);
|
||||
```
|
||||
|
||||
Then, run index.ts
|
||||
|
||||
```sh
|
||||
bun --hot ./index.ts
|
||||
```
|
||||
|
||||
For more information, read the Bun API docs in `node_modules/bun-types/docs/**.mdx`.
|
||||
@@ -0,0 +1,15 @@
|
||||
# mcp-server
|
||||
|
||||
To install dependencies:
|
||||
|
||||
```bash
|
||||
bun install
|
||||
```
|
||||
|
||||
To run:
|
||||
|
||||
```bash
|
||||
bun run index.ts
|
||||
```
|
||||
|
||||
This project was created using `bun init` in bun v1.3.9. [Bun](https://bun.com) is a fast all-in-one JavaScript runtime.
|
||||
@@ -0,0 +1,249 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en"><head>
|
||||
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>Lakehouse — Alerts</title>
|
||||
<style>
|
||||
*{margin:0;padding:0;box-sizing:border-box}
|
||||
body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;color:#b0b8c4;font-size:14px;line-height:1.55;-webkit-font-smoothing:antialiased}
|
||||
a{color:#58a6ff;text-decoration:none}
|
||||
a:hover{color:#79c0ff}
|
||||
|
||||
.bar{background:#0d1117;padding:0 24px;height:56px;border-bottom:1px solid #171d27;display:flex;justify-content:space-between;align-items:center;position:sticky;top:0;z-index:10}
|
||||
.bar h1{font-size:14px;font-weight:600;color:#e6edf3;letter-spacing:-0.2px}
|
||||
.bar nav{display:flex;gap:2px}
|
||||
.bar nav a{font-size:12px;color:#545d68;padding:6px 14px;border-radius:6px;transition:all 0.15s}
|
||||
.bar nav a:hover{color:#e6edf3;background:#161b22}
|
||||
.bar nav a.active{color:#e6edf3;background:#1c2333}
|
||||
|
||||
.wrap{max-width:900px;margin:0 auto;padding:28px 20px 60px}
|
||||
|
||||
h2{color:#e6edf3;font-size:20px;font-weight:700;letter-spacing:-0.3px;margin-bottom:4px}
|
||||
.lede{color:#8b949e;font-size:13px;margin-bottom:18px;line-height:1.6;max-width:680px}
|
||||
|
||||
.card{background:#0d1117;border:1px solid #171d27;border-radius:12px;padding:18px;margin:10px 0}
|
||||
|
||||
.row{display:flex;justify-content:space-between;align-items:center;gap:12px;padding:10px 0;border-bottom:1px solid #171d27}
|
||||
.row:last-child{border-bottom:none}
|
||||
.row .k{color:#8b949e;font-size:12px}
|
||||
.row .v{color:#e6edf3;font-weight:500;font-size:13px;font-family:ui-monospace,Menlo,monospace}
|
||||
|
||||
label{display:block;color:#8b949e;font-size:12px;margin-bottom:6px;margin-top:12px}
|
||||
label:first-child{margin-top:0}
|
||||
input[type=text],input[type=number],input[type=url]{width:100%;padding:10px 14px;background:#161b22;border:1px solid #21262d;border-radius:8px;color:#e6edf3;font-size:13px;outline:none;font-family:ui-monospace,Menlo,monospace}
|
||||
input:focus{border-color:#388bfd}
|
||||
.switch{display:flex;align-items:center;gap:8px;margin-top:4px}
|
||||
.switch input{width:auto;accent-color:#1f6feb}
|
||||
.switch span{color:#c9d1d9;font-size:13px}
|
||||
|
||||
.btn{padding:10px 18px;background:#1f6feb;border:none;border-radius:8px;color:#fff;font-size:13px;font-weight:600;cursor:pointer;margin-right:8px;margin-top:14px}
|
||||
.btn:hover{background:#388bfd}
|
||||
.btn:disabled{opacity:0.4;cursor:wait}
|
||||
.btn.ghost{background:transparent;border:1px solid #21262d;color:#c9d1d9}
|
||||
.btn.ghost:hover{background:#161b22}
|
||||
.btn.green{background:#2ea043}
|
||||
.btn.green:hover{background:#3fb950}
|
||||
|
||||
.entry{background:#161b22;border-radius:6px;padding:10px 14px;margin-bottom:6px;font-size:12px;font-family:ui-monospace,Menlo,monospace;white-space:pre-wrap;color:#c9d1d9;line-height:1.6}
|
||||
.entry .at{color:#545d68;font-size:10px;display:block;margin-bottom:4px}
|
||||
|
||||
.tip{color:#8b949e;font-size:12px;line-height:1.7;padding:10px 14px;border-left:2px solid #21262d;margin:10px 0}
|
||||
.tip strong{color:#c9d1d9}
|
||||
.tip code{background:#161b22;padding:1px 5px;border-radius:3px;font-family:ui-monospace,Menlo,monospace;font-size:11px;color:#79c0ff}
|
||||
|
||||
.result{padding:14px;border-radius:8px;margin-top:10px;font-size:12px;line-height:1.6}
|
||||
.result.ok{background:#0d2818;border:1px solid #2ea04360;color:#86efac}
|
||||
.result.err{background:#3a1a1a;border:1px solid #f8514960;color:#fca5a5}
|
||||
.result.dim{background:#161b22;border:1px solid #21262d;color:#8b949e}
|
||||
|
||||
.chan{display:inline-block;padding:3px 10px;border-radius:10px;font-size:10px;font-weight:600;margin-right:6px;letter-spacing:0.3px}
|
||||
.chan.console{background:#1a2744;color:#79c0ff}
|
||||
.chan.file{background:#1a3a2a;color:#3fb950}
|
||||
.chan.webhook{background:#2a1a3a;color:#bc8cff}
|
||||
|
||||
.footer{border-top:1px solid #171d27;padding:20px;text-align:center;color:#3d444d;font-size:11px}
|
||||
|
||||
@media(max-width:720px){
|
||||
.wrap{padding:20px 12px 40px}
|
||||
.bar nav{display:none}
|
||||
}
|
||||
</style></head>
|
||||
<body>
|
||||
|
||||
<div class="bar">
|
||||
<h1>Lakehouse — Alerts</h1>
|
||||
<nav>
|
||||
<a href=".">Dashboard</a>
|
||||
<a href="console">Walkthrough</a>
|
||||
<a href="proof">Architecture</a>
|
||||
<a href="spec">Spec</a>
|
||||
<a href="onboard">Onboard</a>
|
||||
<a href="alerts" class="active">Alerts</a>
|
||||
<a href="workspaces">Workspaces</a>
|
||||
</nav>
|
||||
</div>
|
||||
|
||||
<div class="wrap">
|
||||
|
||||
<h2>Push alerts — make the system find you</h2>
|
||||
<div class="lede">The daemon runs in the background and dispatches a concise digest whenever something worth notifying changes: a role escalates to tight or critical, a new staffing deadline falls within your warn window, or the playbook memory compounds meaningfully. Phone-first shops don't open dashboards — they need the system to reach out.</div>
|
||||
|
||||
<div class="card">
|
||||
<h3 style="color:#e6edf3;font-size:15px;margin-bottom:10px">Current status</h3>
|
||||
<div id="status"><div style="color:#8b949e">Loading…</div></div>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3 style="color:#e6edf3;font-size:15px;margin-bottom:10px">Configuration</h3>
|
||||
<form id="cfg-form">
|
||||
<label class="switch"><input type="checkbox" id="enabled"> <span>Daemon enabled — fire digests on the interval</span></label>
|
||||
<label>Check interval (minutes)</label>
|
||||
<input type="number" id="interval" min="1" max="1440" value="15">
|
||||
<label>Deadline warn window (days)</label>
|
||||
<input type="number" id="deadline_warn_days" min="1" max="90" value="7">
|
||||
<label>Webhook URL (Slack, Discord, or any POST-accepting endpoint — optional)</label>
|
||||
<input type="url" id="webhook_url" placeholder="https://hooks.slack.com/services/... or https://discord.com/api/webhooks/...">
|
||||
<label>Webhook label (your name for this channel)</label>
|
||||
<input type="text" id="webhook_label" placeholder="e.g. #staffing-ops">
|
||||
<div>
|
||||
<button type="button" class="btn" id="save-btn">Save</button>
|
||||
<button type="button" class="btn green" id="fire-btn">Fire a test digest now</button>
|
||||
</div>
|
||||
<div id="save-result"></div>
|
||||
</form>
|
||||
<div class="tip">
|
||||
<strong>Webhook behavior.</strong> The daemon POSTs JSON <code>{text, digest}</code> to your webhook URL.
|
||||
Slack and Discord both accept this shape if the URL is an incoming-webhook. The <code>text</code>
|
||||
field is pre-formatted human-readable; the <code>digest</code> field is structured for a bot to parse.
|
||||
Interval changes take effect on next server restart.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3 style="color:#e6edf3;font-size:15px;margin-bottom:10px">Recent digests</h3>
|
||||
<div id="recent"><div style="color:#8b949e">Loading…</div></div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<div class="footer">Lakehouse · alerts · daemon log: <code>mcp-server/data/notifications.jsonl</code></div>
|
||||
|
||||
<script>
|
||||
var P=location.pathname.indexOf('/lakehouse')>=0?'/lakehouse':'';
|
||||
var A=location.origin+P;
|
||||
|
||||
function el(t,c,x){var e=document.createElement(t);if(c)e.className=c;if(x!==undefined)e.textContent=String(x);return e}
|
||||
function apiGet(p){return fetch(A+p).then(function(r){return r.json()})}
|
||||
function apiPost(p,b){return fetch(A+p,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(b||{})}).then(function(r){return r.json()})}
|
||||
|
||||
function loadStatus(){
|
||||
apiGet('/alerts/config').then(function(r){
|
||||
var host=document.getElementById('status');host.textContent='';
|
||||
var cfg=r.config||{};
|
||||
var st=r.state||{};
|
||||
var rows=[
|
||||
['Daemon',cfg.enabled?'enabled':'disabled'],
|
||||
['Interval',(cfg.interval_minutes||'?')+' minutes'],
|
||||
['Deadline warn window',(cfg.deadline_warn_days||7)+' days'],
|
||||
['Webhook',cfg.webhook_url?(cfg.webhook_label||'configured'):'none'],
|
||||
['Last run',st.last_run_at?new Date(st.last_run_at).toLocaleString():'never'],
|
||||
];
|
||||
rows.forEach(function(r){
|
||||
var row=el('div','row');
|
||||
row.appendChild(el('span','k',r[0]));
|
||||
row.appendChild(el('span','v',r[1]));
|
||||
host.appendChild(row);
|
||||
});
|
||||
// Populate form
|
||||
document.getElementById('enabled').checked=!!cfg.enabled;
|
||||
document.getElementById('interval').value=cfg.interval_minutes||15;
|
||||
document.getElementById('deadline_warn_days').value=cfg.deadline_warn_days||7;
|
||||
document.getElementById('webhook_url').value=cfg.webhook_url||'';
|
||||
document.getElementById('webhook_label').value=cfg.webhook_label||'';
|
||||
}).catch(function(e){
|
||||
var host=document.getElementById('status');host.textContent='';
|
||||
host.appendChild(el('div',null,'status unavailable: '+(e.message||e)));
|
||||
});
|
||||
}
|
||||
|
||||
function loadRecent(){
|
||||
apiGet('/alerts/recent').then(function(r){
|
||||
var host=document.getElementById('recent');host.textContent='';
|
||||
var entries=r.entries||[];
|
||||
if(entries.length===0){
|
||||
host.appendChild(el('div',null,'No digests yet. The daemon fires every '+(document.getElementById('interval').value||15)+' minutes and only dispatches when something changes. Click "Fire a test digest now" to force one.'));
|
||||
host.firstChild.style.color='#8b949e';
|
||||
return;
|
||||
}
|
||||
entries.forEach(function(e){
|
||||
var card=el('div','entry');
|
||||
if(e.at) card.appendChild(el('span','at',new Date(e.at).toLocaleString()));
|
||||
card.appendChild(document.createTextNode(e.text||JSON.stringify(e.digest)));
|
||||
host.appendChild(card);
|
||||
});
|
||||
}).catch(function(e){
|
||||
var host=document.getElementById('recent');host.textContent='';
|
||||
host.appendChild(el('div',null,'recent digests unavailable: '+(e.message||e)));
|
||||
});
|
||||
}
|
||||
|
||||
function save(){
|
||||
var btn=document.getElementById('save-btn');
|
||||
btn.disabled=true;
|
||||
var body={
|
||||
enabled:document.getElementById('enabled').checked,
|
||||
interval_minutes:Number(document.getElementById('interval').value)||15,
|
||||
deadline_warn_days:Number(document.getElementById('deadline_warn_days').value)||7,
|
||||
webhook_url:document.getElementById('webhook_url').value.trim(),
|
||||
webhook_label:document.getElementById('webhook_label').value.trim(),
|
||||
};
|
||||
apiPost('/alerts/config',body).then(function(r){
|
||||
btn.disabled=false;
|
||||
var out=document.getElementById('save-result');out.textContent='';
|
||||
if(r.saved){
|
||||
var ok=el('div','result ok',r.note||'saved.');out.appendChild(ok);
|
||||
loadStatus();
|
||||
} else {
|
||||
out.appendChild(el('div','result err',JSON.stringify(r)));
|
||||
}
|
||||
}).catch(function(e){
|
||||
btn.disabled=false;
|
||||
var out=document.getElementById('save-result');out.textContent='';
|
||||
out.appendChild(el('div','result err','save failed: '+(e.message||e)));
|
||||
});
|
||||
}
|
||||
|
||||
function fire(){
|
||||
var btn=document.getElementById('fire-btn');
|
||||
btn.disabled=true;btn.textContent='Firing…';
|
||||
apiPost('/alerts/fire',{}).then(function(r){
|
||||
btn.disabled=false;btn.textContent='Fire a test digest now';
|
||||
var out=document.getElementById('save-result');out.textContent='';
|
||||
if(!r.fired){
|
||||
out.appendChild(el('div','result dim',r.reason||'nothing to dispatch'));
|
||||
} else {
|
||||
var ok=el('div','result ok');
|
||||
ok.appendChild(document.createTextNode('Dispatched to: '));
|
||||
(r.channels||[]).forEach(function(c){
|
||||
var chip=el('span','chan '+c,c);ok.appendChild(chip);
|
||||
});
|
||||
if(r.errors&&r.errors.length){
|
||||
ok.appendChild(document.createElement('br'));
|
||||
ok.appendChild(document.createTextNode('errors: '+r.errors.join(' | ')));
|
||||
}
|
||||
out.appendChild(ok);
|
||||
setTimeout(loadRecent,500);
|
||||
}
|
||||
}).catch(function(e){
|
||||
btn.disabled=false;btn.textContent='Fire a test digest now';
|
||||
var out=document.getElementById('save-result');out.textContent='';
|
||||
out.appendChild(el('div','result err','fire failed: '+(e.message||e)));
|
||||
});
|
||||
}
|
||||
|
||||
document.addEventListener('DOMContentLoaded',function(){
|
||||
loadStatus();loadRecent();
|
||||
document.getElementById('save-btn').addEventListener('click',save);
|
||||
document.getElementById('fire-btn').addEventListener('click',fire);
|
||||
});
|
||||
</script>
|
||||
</body></html>
|
||||
@@ -0,0 +1,211 @@
|
||||
{
|
||||
"lockfileVersion": 1,
|
||||
"configVersion": 1,
|
||||
"workspaces": {
|
||||
"": {
|
||||
"name": "mcp-server",
|
||||
"dependencies": {
|
||||
"@modelcontextprotocol/sdk": "^1.29.0",
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/bun": "latest",
|
||||
},
|
||||
"peerDependencies": {
|
||||
"typescript": "^5",
|
||||
},
|
||||
},
|
||||
},
|
||||
"packages": {
|
||||
"@hono/node-server": ["@hono/node-server@1.19.14", "", { "peerDependencies": { "hono": "^4" } }, "sha512-GwtvgtXxnWsucXvbQXkRgqksiH2Qed37H9xHZocE5sA3N8O8O8/8FA3uclQXxXVzc9XBZuEOMK7+r02FmSpHtw=="],
|
||||
|
||||
"@modelcontextprotocol/sdk": ["@modelcontextprotocol/sdk@1.29.0", "", { "dependencies": { "@hono/node-server": "^1.19.9", "ajv": "^8.17.1", "ajv-formats": "^3.0.1", "content-type": "^1.0.5", "cors": "^2.8.5", "cross-spawn": "^7.0.5", "eventsource": "^3.0.2", "eventsource-parser": "^3.0.0", "express": "^5.2.1", "express-rate-limit": "^8.2.1", "hono": "^4.11.4", "jose": "^6.1.3", "json-schema-typed": "^8.0.2", "pkce-challenge": "^5.0.0", "raw-body": "^3.0.0", "zod": "^3.25 || ^4.0", "zod-to-json-schema": "^3.25.1" }, "peerDependencies": { "@cfworker/json-schema": "^4.1.1" }, "optionalPeers": ["@cfworker/json-schema"] }, "sha512-zo37mZA9hJWpULgkRpowewez1y6ML5GsXJPY8FI0tBBCd77HEvza4jDqRKOXgHNn867PVGCyTdzqpz0izu5ZjQ=="],
|
||||
|
||||
"@types/bun": ["@types/bun@1.3.12", "", { "dependencies": { "bun-types": "1.3.12" } }, "sha512-DBv81elK+/VSwXHDlnH3Qduw+KxkTIWi7TXkAeh24zpi5l0B2kUg9Ga3tb4nJaPcOFswflgi/yAvMVBPrxMB+A=="],
|
||||
|
||||
"@types/node": ["@types/node@25.6.0", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],
|
||||
|
||||
"accepts": ["accepts@2.0.0", "", { "dependencies": { "mime-types": "^3.0.0", "negotiator": "^1.0.0" } }, "sha512-5cvg6CtKwfgdmVqY1WIiXKc3Q1bkRqGLi+2W/6ao+6Y7gu/RCwRuAhGEzh5B4KlszSuTLgZYuqFqo5bImjNKng=="],
|
||||
|
||||
"ajv": ["ajv@8.18.0", "", { "dependencies": { "fast-deep-equal": "^3.1.3", "fast-uri": "^3.0.1", "json-schema-traverse": "^1.0.0", "require-from-string": "^2.0.2" } }, "sha512-PlXPeEWMXMZ7sPYOHqmDyCJzcfNrUr3fGNKtezX14ykXOEIvyK81d+qydx89KY5O71FKMPaQ2vBfBFI5NHR63A=="],
|
||||
|
||||
"ajv-formats": ["ajv-formats@3.0.1", "", { "dependencies": { "ajv": "^8.0.0" } }, "sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ=="],
|
||||
|
||||
"body-parser": ["body-parser@2.2.2", "", { "dependencies": { "bytes": "^3.1.2", "content-type": "^1.0.5", "debug": "^4.4.3", "http-errors": "^2.0.0", "iconv-lite": "^0.7.0", "on-finished": "^2.4.1", "qs": "^6.14.1", "raw-body": "^3.0.1", "type-is": "^2.0.1" } }, "sha512-oP5VkATKlNwcgvxi0vM0p/D3n2C3EReYVX+DNYs5TjZFn/oQt2j+4sVJtSMr18pdRr8wjTcBl6LoV+FUwzPmNA=="],
|
||||
|
||||
"bun-types": ["bun-types@1.3.12", "", { "dependencies": { "@types/node": "*" } }, "sha512-HqOLj5PoFajAQciOMRiIZGNoKxDJSr6qigAttOX40vJuSp6DN/CxWp9s3C1Xwm4oH7ybueITwiaOcWXoYVoRkA=="],
|
||||
|
||||
"bytes": ["bytes@3.1.2", "", {}, "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg=="],
|
||||
|
||||
"call-bind-apply-helpers": ["call-bind-apply-helpers@1.0.2", "", { "dependencies": { "es-errors": "^1.3.0", "function-bind": "^1.1.2" } }, "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ=="],
|
||||
|
||||
"call-bound": ["call-bound@1.0.4", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.2", "get-intrinsic": "^1.3.0" } }, "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg=="],
|
||||
|
||||
"content-disposition": ["content-disposition@1.1.0", "", {}, "sha512-5jRCH9Z/+DRP7rkvY83B+yGIGX96OYdJmzngqnw2SBSxqCFPd0w2km3s5iawpGX8krnwSGmF0FW5Nhr0Hfai3g=="],
|
||||
|
||||
"content-type": ["content-type@1.0.5", "", {}, "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA=="],
|
||||
|
||||
"cookie": ["cookie@0.7.2", "", {}, "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w=="],
|
||||
|
||||
"cookie-signature": ["cookie-signature@1.2.2", "", {}, "sha512-D76uU73ulSXrD1UXF4KE2TMxVVwhsnCgfAyTg9k8P6KGZjlXKrOLe4dJQKI3Bxi5wjesZoFXJWElNWBjPZMbhg=="],
|
||||
|
||||
"cors": ["cors@2.8.6", "", { "dependencies": { "object-assign": "^4", "vary": "^1" } }, "sha512-tJtZBBHA6vjIAaF6EnIaq6laBBP9aq/Y3ouVJjEfoHbRBcHBAHYcMh/w8LDrk2PvIMMq8gmopa5D4V8RmbrxGw=="],
|
||||
|
||||
"cross-spawn": ["cross-spawn@7.0.6", "", { "dependencies": { "path-key": "^3.1.0", "shebang-command": "^2.0.0", "which": "^2.0.1" } }, "sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA=="],
|
||||
|
||||
"debug": ["debug@4.4.3", "", { "dependencies": { "ms": "^2.1.3" } }, "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA=="],
|
||||
|
||||
"depd": ["depd@2.0.0", "", {}, "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw=="],
|
||||
|
||||
"dunder-proto": ["dunder-proto@1.0.1", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.1", "es-errors": "^1.3.0", "gopd": "^1.2.0" } }, "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A=="],
|
||||
|
||||
"ee-first": ["ee-first@1.1.1", "", {}, "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow=="],
|
||||
|
||||
"encodeurl": ["encodeurl@2.0.0", "", {}, "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg=="],
|
||||
|
||||
"es-define-property": ["es-define-property@1.0.1", "", {}, "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g=="],
|
||||
|
||||
"es-errors": ["es-errors@1.3.0", "", {}, "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw=="],
|
||||
|
||||
"es-object-atoms": ["es-object-atoms@1.1.1", "", { "dependencies": { "es-errors": "^1.3.0" } }, "sha512-FGgH2h8zKNim9ljj7dankFPcICIK9Cp5bm+c2gQSYePhpaG5+esrLODihIorn+Pe6FGJzWhXQotPv73jTaldXA=="],
|
||||
|
||||
"escape-html": ["escape-html@1.0.3", "", {}, "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow=="],
|
||||
|
||||
"etag": ["etag@1.8.1", "", {}, "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg=="],
|
||||
|
||||
"eventsource": ["eventsource@3.0.7", "", { "dependencies": { "eventsource-parser": "^3.0.1" } }, "sha512-CRT1WTyuQoD771GW56XEZFQ/ZoSfWid1alKGDYMmkt2yl8UXrVR4pspqWNEcqKvVIzg6PAltWjxcSSPrboA4iA=="],
|
||||
|
||||
"eventsource-parser": ["eventsource-parser@3.0.6", "", {}, "sha512-Vo1ab+QXPzZ4tCa8SwIHJFaSzy4R6SHf7BY79rFBDf0idraZWAkYrDjDj8uWaSm3S2TK+hJ7/t1CEmZ7jXw+pg=="],
|
||||
|
||||
"express": ["express@5.2.1", "", { "dependencies": { "accepts": "^2.0.0", "body-parser": "^2.2.1", "content-disposition": "^1.0.0", "content-type": "^1.0.5", "cookie": "^0.7.1", "cookie-signature": "^1.2.1", "debug": "^4.4.0", "depd": "^2.0.0", "encodeurl": "^2.0.0", "escape-html": "^1.0.3", "etag": "^1.8.1", "finalhandler": "^2.1.0", "fresh": "^2.0.0", "http-errors": "^2.0.0", "merge-descriptors": "^2.0.0", "mime-types": "^3.0.0", "on-finished": "^2.4.1", "once": "^1.4.0", "parseurl": "^1.3.3", "proxy-addr": "^2.0.7", "qs": "^6.14.0", "range-parser": "^1.2.1", "router": "^2.2.0", "send": "^1.1.0", "serve-static": "^2.2.0", "statuses": "^2.0.1", "type-is": "^2.0.1", "vary": "^1.1.2" } }, "sha512-hIS4idWWai69NezIdRt2xFVofaF4j+6INOpJlVOLDO8zXGpUVEVzIYk12UUi2JzjEzWL3IOAxcTubgz9Po0yXw=="],
|
||||
|
||||
"express-rate-limit": ["express-rate-limit@8.3.2", "", { "dependencies": { "ip-address": "10.1.0" }, "peerDependencies": { "express": ">= 4.11" } }, "sha512-77VmFeJkO0/rvimEDuUC5H30oqUC4EyOhyGccfqoLebB0oiEYfM7nwPrsDsBL1gsTpwfzX8SFy2MT3TDyRq+bg=="],
|
||||
|
||||
"fast-deep-equal": ["fast-deep-equal@3.1.3", "", {}, "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q=="],
|
||||
|
||||
"fast-uri": ["fast-uri@3.1.0", "", {}, "sha512-iPeeDKJSWf4IEOasVVrknXpaBV0IApz/gp7S2bb7Z4Lljbl2MGJRqInZiUrQwV16cpzw/D3S5j5Julj/gT52AA=="],
|
||||
|
||||
"finalhandler": ["finalhandler@2.1.1", "", { "dependencies": { "debug": "^4.4.0", "encodeurl": "^2.0.0", "escape-html": "^1.0.3", "on-finished": "^2.4.1", "parseurl": "^1.3.3", "statuses": "^2.0.1" } }, "sha512-S8KoZgRZN+a5rNwqTxlZZePjT/4cnm0ROV70LedRHZ0p8u9fRID0hJUZQpkKLzro8LfmC8sx23bY6tVNxv8pQA=="],
|
||||
|
||||
"forwarded": ["forwarded@0.2.0", "", {}, "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow=="],
|
||||
|
||||
"fresh": ["fresh@2.0.0", "", {}, "sha512-Rx/WycZ60HOaqLKAi6cHRKKI7zxWbJ31MhntmtwMoaTeF7XFH9hhBp8vITaMidfljRQ6eYWCKkaTK+ykVJHP2A=="],
|
||||
|
||||
"function-bind": ["function-bind@1.1.2", "", {}, "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA=="],
|
||||
|
||||
"get-intrinsic": ["get-intrinsic@1.3.0", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.2", "es-define-property": "^1.0.1", "es-errors": "^1.3.0", "es-object-atoms": "^1.1.1", "function-bind": "^1.1.2", "get-proto": "^1.0.1", "gopd": "^1.2.0", "has-symbols": "^1.1.0", "hasown": "^2.0.2", "math-intrinsics": "^1.1.0" } }, "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ=="],
|
||||
|
||||
"get-proto": ["get-proto@1.0.1", "", { "dependencies": { "dunder-proto": "^1.0.1", "es-object-atoms": "^1.0.0" } }, "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g=="],
|
||||
|
||||
"gopd": ["gopd@1.2.0", "", {}, "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg=="],
|
||||
|
||||
"has-symbols": ["has-symbols@1.1.0", "", {}, "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ=="],
|
||||
|
||||
"hasown": ["hasown@2.0.2", "", { "dependencies": { "function-bind": "^1.1.2" } }, "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ=="],
|
||||
|
||||
"hono": ["hono@4.12.14", "", {}, "sha512-am5zfg3yu6sqn5yjKBNqhnTX7Cv+m00ox+7jbaKkrLMRJ4rAdldd1xPd/JzbBWspqaQv6RSTrgFN95EsfhC+7w=="],
|
||||
|
||||
"http-errors": ["http-errors@2.0.1", "", { "dependencies": { "depd": "~2.0.0", "inherits": "~2.0.4", "setprototypeof": "~1.2.0", "statuses": "~2.0.2", "toidentifier": "~1.0.1" } }, "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ=="],
|
||||
|
||||
"iconv-lite": ["iconv-lite@0.7.2", "", { "dependencies": { "safer-buffer": ">= 2.1.2 < 3.0.0" } }, "sha512-im9DjEDQ55s9fL4EYzOAv0yMqmMBSZp6G0VvFyTMPKWxiSBHUj9NW/qqLmXUwXrrM7AvqSlTCfvqRb0cM8yYqw=="],
|
||||
|
||||
"inherits": ["inherits@2.0.4", "", {}, "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="],
|
||||
|
||||
"ip-address": ["ip-address@10.1.0", "", {}, "sha512-XXADHxXmvT9+CRxhXg56LJovE+bmWnEWB78LB83VZTprKTmaC5QfruXocxzTZ2Kl0DNwKuBdlIhjL8LeY8Sf8Q=="],
|
||||
|
||||
"ipaddr.js": ["ipaddr.js@1.9.1", "", {}, "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g=="],
|
||||
|
||||
"is-promise": ["is-promise@4.0.0", "", {}, "sha512-hvpoI6korhJMnej285dSg6nu1+e6uxs7zG3BYAm5byqDsgJNWwxzM6z6iZiAgQR4TJ30JmBTOwqZUw3WlyH3AQ=="],
|
||||
|
||||
"isexe": ["isexe@2.0.0", "", {}, "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw=="],
|
||||
|
||||
"jose": ["jose@6.2.2", "", {}, "sha512-d7kPDd34KO/YnzaDOlikGpOurfF0ByC2sEV4cANCtdqLlTfBlw2p14O/5d/zv40gJPbIQxfES3nSx1/oYNyuZQ=="],
|
||||
|
||||
"json-schema-traverse": ["json-schema-traverse@1.0.0", "", {}, "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug=="],
|
||||
|
||||
"json-schema-typed": ["json-schema-typed@8.0.2", "", {}, "sha512-fQhoXdcvc3V28x7C7BMs4P5+kNlgUURe2jmUT1T//oBRMDrqy1QPelJimwZGo7Hg9VPV3EQV5Bnq4hbFy2vetA=="],
|
||||
|
||||
"math-intrinsics": ["math-intrinsics@1.1.0", "", {}, "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g=="],
|
||||
|
||||
"media-typer": ["media-typer@1.1.0", "", {}, "sha512-aisnrDP4GNe06UcKFnV5bfMNPBUw4jsLGaWwWfnH3v02GnBuXX2MCVn5RbrWo0j3pczUilYblq7fQ7Nw2t5XKw=="],
|
||||
|
||||
"merge-descriptors": ["merge-descriptors@2.0.0", "", {}, "sha512-Snk314V5ayFLhp3fkUREub6WtjBfPdCPY1Ln8/8munuLuiYhsABgBVWsozAG+MWMbVEvcdcpbi9R7ww22l9Q3g=="],
|
||||
|
||||
"mime-db": ["mime-db@1.54.0", "", {}, "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ=="],
|
||||
|
||||
"mime-types": ["mime-types@3.0.2", "", { "dependencies": { "mime-db": "^1.54.0" } }, "sha512-Lbgzdk0h4juoQ9fCKXW4by0UJqj+nOOrI9MJ1sSj4nI8aI2eo1qmvQEie4VD1glsS250n15LsWsYtCugiStS5A=="],
|
||||
|
||||
"ms": ["ms@2.1.3", "", {}, "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="],
|
||||
|
||||
"negotiator": ["negotiator@1.0.0", "", {}, "sha512-8Ofs/AUQh8MaEcrlq5xOX0CQ9ypTF5dl78mjlMNfOK08fzpgTHQRQPBxcPlEtIw0yRpws+Zo/3r+5WRby7u3Gg=="],
|
||||
|
||||
"object-assign": ["object-assign@4.1.1", "", {}, "sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg=="],
|
||||
|
||||
"object-inspect": ["object-inspect@1.13.4", "", {}, "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew=="],
|
||||
|
||||
"on-finished": ["on-finished@2.4.1", "", { "dependencies": { "ee-first": "1.1.1" } }, "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg=="],
|
||||
|
||||
"once": ["once@1.4.0", "", { "dependencies": { "wrappy": "1" } }, "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w=="],
|
||||
|
||||
"parseurl": ["parseurl@1.3.3", "", {}, "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ=="],
|
||||
|
||||
"path-key": ["path-key@3.1.1", "", {}, "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q=="],
|
||||
|
||||
"path-to-regexp": ["path-to-regexp@8.4.2", "", {}, "sha512-qRcuIdP69NPm4qbACK+aDogI5CBDMi1jKe0ry5rSQJz8JVLsC7jV8XpiJjGRLLol3N+R5ihGYcrPLTno6pAdBA=="],
|
||||
|
||||
"pkce-challenge": ["pkce-challenge@5.0.1", "", {}, "sha512-wQ0b/W4Fr01qtpHlqSqspcj3EhBvimsdh0KlHhH8HRZnMsEa0ea2fTULOXOS9ccQr3om+GcGRk4e+isrZWV8qQ=="],
|
||||
|
||||
"proxy-addr": ["proxy-addr@2.0.7", "", { "dependencies": { "forwarded": "0.2.0", "ipaddr.js": "1.9.1" } }, "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg=="],
|
||||
|
||||
"qs": ["qs@6.15.1", "", { "dependencies": { "side-channel": "^1.1.0" } }, "sha512-6YHEFRL9mfgcAvql/XhwTvf5jKcOiiupt2FiJxHkiX1z4j7WL8J/jRHYLluORvc1XxB5rV20KoeK00gVJamspg=="],
|
||||
|
||||
"range-parser": ["range-parser@1.2.1", "", {}, "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg=="],
|
||||
|
||||
"raw-body": ["raw-body@3.0.2", "", { "dependencies": { "bytes": "~3.1.2", "http-errors": "~2.0.1", "iconv-lite": "~0.7.0", "unpipe": "~1.0.0" } }, "sha512-K5zQjDllxWkf7Z5xJdV0/B0WTNqx6vxG70zJE4N0kBs4LovmEYWJzQGxC9bS9RAKu3bgM40lrd5zoLJ12MQ5BA=="],
|
||||
|
||||
"require-from-string": ["require-from-string@2.0.2", "", {}, "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw=="],
|
||||
|
||||
"router": ["router@2.2.0", "", { "dependencies": { "debug": "^4.4.0", "depd": "^2.0.0", "is-promise": "^4.0.0", "parseurl": "^1.3.3", "path-to-regexp": "^8.0.0" } }, "sha512-nLTrUKm2UyiL7rlhapu/Zl45FwNgkZGaCpZbIHajDYgwlJCOzLSk+cIPAnsEqV955GjILJnKbdQC1nVPz+gAYQ=="],
|
||||
|
||||
"safer-buffer": ["safer-buffer@2.1.2", "", {}, "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="],
|
||||
|
||||
"send": ["send@1.2.1", "", { "dependencies": { "debug": "^4.4.3", "encodeurl": "^2.0.0", "escape-html": "^1.0.3", "etag": "^1.8.1", "fresh": "^2.0.0", "http-errors": "^2.0.1", "mime-types": "^3.0.2", "ms": "^2.1.3", "on-finished": "^2.4.1", "range-parser": "^1.2.1", "statuses": "^2.0.2" } }, "sha512-1gnZf7DFcoIcajTjTwjwuDjzuz4PPcY2StKPlsGAQ1+YH20IRVrBaXSWmdjowTJ6u8Rc01PoYOGHXfP1mYcZNQ=="],
|
||||
|
||||
"serve-static": ["serve-static@2.2.1", "", { "dependencies": { "encodeurl": "^2.0.0", "escape-html": "^1.0.3", "parseurl": "^1.3.3", "send": "^1.2.0" } }, "sha512-xRXBn0pPqQTVQiC8wyQrKs2MOlX24zQ0POGaj0kultvoOCstBQM5yvOhAVSUwOMjQtTvsPWoNCHfPGwaaQJhTw=="],
|
||||
|
||||
"setprototypeof": ["setprototypeof@1.2.0", "", {}, "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw=="],
|
||||
|
||||
"shebang-command": ["shebang-command@2.0.0", "", { "dependencies": { "shebang-regex": "^3.0.0" } }, "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA=="],
|
||||
|
||||
"shebang-regex": ["shebang-regex@3.0.0", "", {}, "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A=="],
|
||||
|
||||
"side-channel": ["side-channel@1.1.0", "", { "dependencies": { "es-errors": "^1.3.0", "object-inspect": "^1.13.3", "side-channel-list": "^1.0.0", "side-channel-map": "^1.0.1", "side-channel-weakmap": "^1.0.2" } }, "sha512-ZX99e6tRweoUXqR+VBrslhda51Nh5MTQwou5tnUDgbtyM0dBgmhEDtWGP/xbKn6hqfPRHujUNwz5fy/wbbhnpw=="],
|
||||
|
||||
"side-channel-list": ["side-channel-list@1.0.1", "", { "dependencies": { "es-errors": "^1.3.0", "object-inspect": "^1.13.4" } }, "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w=="],
|
||||
|
||||
"side-channel-map": ["side-channel-map@1.0.1", "", { "dependencies": { "call-bound": "^1.0.2", "es-errors": "^1.3.0", "get-intrinsic": "^1.2.5", "object-inspect": "^1.13.3" } }, "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA=="],
|
||||
|
||||
"side-channel-weakmap": ["side-channel-weakmap@1.0.2", "", { "dependencies": { "call-bound": "^1.0.2", "es-errors": "^1.3.0", "get-intrinsic": "^1.2.5", "object-inspect": "^1.13.3", "side-channel-map": "^1.0.1" } }, "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A=="],
|
||||
|
||||
"statuses": ["statuses@2.0.2", "", {}, "sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw=="],
|
||||
|
||||
"toidentifier": ["toidentifier@1.0.1", "", {}, "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA=="],
|
||||
|
||||
"type-is": ["type-is@2.0.1", "", { "dependencies": { "content-type": "^1.0.5", "media-typer": "^1.1.0", "mime-types": "^3.0.0" } }, "sha512-OZs6gsjF4vMp32qrCbiVSkrFmXtG/AZhY3t0iAMrMBiAZyV9oALtXO8hsrHbMXF9x6L3grlFuwW2oAz7cav+Gw=="],
|
||||
|
||||
"typescript": ["typescript@5.9.3", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="],
|
||||
|
||||
"undici-types": ["undici-types@7.19.2", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],
|
||||
|
||||
"unpipe": ["unpipe@1.0.0", "", {}, "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ=="],
|
||||
|
||||
"vary": ["vary@1.1.2", "", {}, "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg=="],
|
||||
|
||||
"which": ["which@2.0.2", "", { "dependencies": { "isexe": "^2.0.0" }, "bin": { "node-which": "./bin/node-which" } }, "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA=="],
|
||||
|
||||
"wrappy": ["wrappy@1.0.2", "", {}, "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ=="],
|
||||
|
||||
"zod": ["zod@4.3.6", "", {}, "sha512-rftlrkhHZOcjDwkGlnUtZZkvaPHCsDATp4pGpuOOMDaTdDDXF91wuVDJoWoPsKX/3YPQ5fHuF3STjcYyKr+Qhg=="],
|
||||
|
||||
"zod-to-json-schema": ["zod-to-json-schema@3.25.2", "", { "peerDependencies": { "zod": "^3.25.28 || ^4" } }, "sha512-O/PgfnpT1xKSDeQYSCfRI5Gy3hPf91mKVDuYLUHZJMiDFptvP41MSnWofm8dnCm0256ZNfZIM7DSzuSMAFnjHA=="],
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,470 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en"><head>
|
||||
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>Lakehouse — What Your Staffing System Would Do</title>
|
||||
<style>
|
||||
*{margin:0;padding:0;box-sizing:border-box}
|
||||
body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;color:#b0b8c4;font-size:14px;line-height:1.55;-webkit-font-smoothing:antialiased}
|
||||
a{color:#58a6ff;text-decoration:none}
|
||||
a:hover{color:#79c0ff}
|
||||
|
||||
.bar{background:#0d1117;padding:0 24px;height:56px;border-bottom:1px solid #171d27;display:flex;justify-content:space-between;align-items:center;position:sticky;top:0;z-index:10}
|
||||
.bar h1{font-size:14px;font-weight:600;color:#e6edf3;letter-spacing:-0.2px}
|
||||
.bar nav{display:flex;gap:2px}
|
||||
.bar nav a{font-size:12px;color:#545d68;padding:6px 14px;border-radius:6px;transition:all 0.15s}
|
||||
.bar nav a:hover{color:#e6edf3;background:#161b22}
|
||||
.bar nav a.active{color:#e6edf3;background:#1c2333}
|
||||
.bar .rt{font-size:11px;color:#545d68}
|
||||
|
||||
.wrap{max-width:1040px;margin:0 auto;padding:28px 20px 60px}
|
||||
|
||||
.chapter{margin-bottom:48px}
|
||||
.chapter .num{color:#545d68;font-size:11px;font-weight:600;letter-spacing:1.6px;text-transform:uppercase;margin-bottom:6px}
|
||||
.chapter h2{color:#e6edf3;font-size:24px;font-weight:700;letter-spacing:-0.4px;margin-bottom:8px;line-height:1.2}
|
||||
.chapter .lede{color:#8b949e;font-size:14px;margin-bottom:18px;max-width:680px;line-height:1.6}
|
||||
|
||||
.card{background:#0d1117;border:1px solid #171d27;border-radius:12px;padding:20px;margin-bottom:12px}
|
||||
.grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(240px,1fr));gap:10px}
|
||||
|
||||
.stat-lg{padding:18px 20px}
|
||||
.stat-lg .n{font-size:30px;font-weight:800;color:#e6edf3;letter-spacing:-1px;line-height:1}
|
||||
.stat-lg .l{font-size:10px;color:#545d68;text-transform:uppercase;letter-spacing:1.2px;margin-top:8px;font-weight:600}
|
||||
.stat-lg .sub{font-size:12px;color:#8b949e;margin-top:4px}
|
||||
|
||||
.pill{display:inline-block;padding:3px 10px;border-radius:10px;font-size:10px;font-weight:600;letter-spacing:0.3px;margin-right:6px}
|
||||
.pill.real{background:#0d2818;color:#3fb950;border:1px solid #2ea04380}
|
||||
.pill.synth{background:#1a1a2e;color:#bc8cff;border:1px solid #5530988c}
|
||||
.pill.flow{background:#0d2340;color:#58a6ff;border:1px solid #1f6feb80}
|
||||
|
||||
.row{display:flex;justify-content:space-between;align-items:center;gap:12px;padding:10px 14px;background:#0d1117;border:1px solid #171d27;border-radius:8px;margin-bottom:6px;font-size:13px}
|
||||
.row:hover{border-color:#21262d}
|
||||
.row .title{color:#e6edf3;font-weight:500}
|
||||
.row .meta{color:#8b949e;font-size:11px;margin-top:2px}
|
||||
.row .val{color:#58a6ff;font-weight:600;white-space:nowrap}
|
||||
|
||||
details{background:#0d1117;border:1px solid #171d27;border-radius:8px;padding:10px 14px;margin-bottom:6px}
|
||||
details summary{cursor:pointer;font-size:13px;color:#e6edf3;font-weight:500;list-style:none}
|
||||
details summary::-webkit-details-marker{display:none}
|
||||
details .body{padding-top:10px;font-size:12px;color:#8b949e}
|
||||
|
||||
.accent-l{border-left:3px solid #2ea043}
|
||||
.accent-b{border-left:3px solid #1f6feb}
|
||||
.accent-a{border-left:3px solid #bc8cff}
|
||||
.accent-w{border-left:3px solid #d29922}
|
||||
|
||||
.worker{display:flex;align-items:center;gap:10px;padding:8px 10px;background:#161b22;border-radius:6px;margin-bottom:4px;font-size:12px}
|
||||
.worker .av{width:28px;height:28px;border-radius:6px;background:#1a2744;display:flex;align-items:center;justify-content:center;font-weight:600;color:#e6edf3;font-size:10px;flex-shrink:0}
|
||||
.worker .info{flex:1;min-width:0}
|
||||
.worker .nm{color:#e6edf3;font-weight:500}
|
||||
.worker .why{color:#545d68;font-size:11px;margin-top:1px}
|
||||
.worker .score{color:#58a6ff;font-size:11px;font-weight:600;white-space:nowrap}
|
||||
|
||||
.mem-chip{background:#0d2818;border:1px solid #2ea04360;border-radius:6px;padding:8px 12px;font-size:11px;color:#86efac;line-height:1.5;margin-top:6px}
|
||||
.mem-chip .l{color:#3fb950;font-weight:600;margin-right:6px}
|
||||
|
||||
.boost-chip{display:inline-block;margin-left:6px;padding:2px 7px;border-radius:9px;font-size:9px;font-weight:600;background:#0d2818;border:1px solid #2ea043;color:#3fb950;vertical-align:middle}
|
||||
|
||||
.try-box{background:#0d1117;border:1px solid #171d27;border-radius:12px;padding:16px}
|
||||
.try-box input{width:100%;padding:12px 16px;background:#161b22;border:1px solid #21262d;border-radius:8px;color:#e6edf3;font-size:13px;outline:none;margin-bottom:10px}
|
||||
.try-box input:focus{border-color:#388bfd}
|
||||
.try-box button{padding:10px 20px;background:#1f6feb;border:none;border-radius:8px;color:#fff;font-size:13px;font-weight:600;cursor:pointer}
|
||||
.try-box button:hover{background:#388bfd}
|
||||
.try-box button:disabled{opacity:0.5;cursor:wait}
|
||||
|
||||
.footer{border-top:1px solid #171d27;padding:20px;text-align:center;color:#3d444d;font-size:11px}
|
||||
|
||||
.loading{color:#484f58;font-style:italic;padding:20px 0;text-align:center}
|
||||
.err{color:#f85149;font-size:12px;padding:10px}
|
||||
|
||||
.narr{color:#8b949e;font-size:13px;line-height:1.7;margin:10px 0;padding:10px 14px;border-left:2px solid #21262d}
|
||||
.narr strong{color:#c9d1d9;font-weight:600}
|
||||
|
||||
.step-label{color:#58a6ff;font-size:12px;margin-top:12px;font-weight:600}
|
||||
.step-body{color:#c9d1d9;font-size:13px;margin-top:2px}
|
||||
|
||||
@media(max-width:720px){
|
||||
.wrap{padding:20px 12px 40px}
|
||||
.chapter h2{font-size:20px}
|
||||
.bar nav{display:none}
|
||||
}
|
||||
</style></head>
|
||||
<body>
|
||||
|
||||
<div class="bar">
|
||||
<h1>Lakehouse — What Your Staffing System Would Do</h1>
|
||||
<nav>
|
||||
<a href=".">Dashboard</a>
|
||||
<a href="console" class="active">Walkthrough</a>
|
||||
<a href="proof">Architecture</a>
|
||||
<a href="spec">Spec</a>
|
||||
<a href="onboard">Onboard</a>
|
||||
<a href="alerts">Alerts</a>
|
||||
<a href="workspaces">Workspaces</a>
|
||||
</nav>
|
||||
<div class="rt" id="hdr-time">Reading live state…</div>
|
||||
</div>
|
||||
|
||||
<div class="wrap">
|
||||
|
||||
<div class="chapter">
|
||||
<div class="num">Chapter 1</div>
|
||||
<h2>Right now, this system is already thinking</h2>
|
||||
<div class="lede">Before you touched anything, it pulled real Chicago building-permit data, measured demand, checked your bench, and began flagging roles that need attention. This isn't theoretical — open your browser network tab and watch the fetches land.</div>
|
||||
<div class="grid" id="ch1-stats"><div class="loading">Fetching live state…</div></div>
|
||||
<div class="narr" id="ch1-narr"></div>
|
||||
</div>
|
||||
|
||||
<div class="chapter">
|
||||
<div class="num">Chapter 2</div>
|
||||
<h2>The demand signal is real, not made up</h2>
|
||||
<div class="lede">Chicago's Department of Buildings publishes every permit they issue. Below are the largest categories of construction filed in the last 30 days. If a staffer doesn't believe our numbers, they can verify at <a href="https://data.cityofchicago.org/Buildings/Building-Permits/ydr8-5enu" target="_blank" rel="noopener">data.cityofchicago.org</a>.</div>
|
||||
<div id="ch2-permits"><div class="loading">Loading permit feed…</div></div>
|
||||
</div>
|
||||
|
||||
<div class="chapter">
|
||||
<div class="num">Chapter 3</div>
|
||||
<h2>Where your own data would live</h2>
|
||||
<div class="lede">The system stores data in labeled catalogs. Purple pills = synthetic stand-ins you'd swap for your real ATS/CRM/call-log exports. Blue pills = data the system generates about itself (playbooks, audit trails). Nothing else in the pipeline changes — only the source.</div>
|
||||
<div id="ch3-datasets"><div class="loading">Enumerating catalog…</div></div>
|
||||
<div class="narr">
|
||||
<strong>The swap path.</strong> workers_500k → your ATS export (same schema shape). candidates → your CRM. call_log → your phone system's CDR. timesheets → your payroll export. Once ingested, every behavior you see on the dashboard applies to your real data. No re-training. No replatform.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="chapter">
|
||||
<div class="num">Chapter 4</div>
|
||||
<h2>Watch the system rank candidates in real time</h2>
|
||||
<div class="lede">This takes the most recent Chicago permit, derives the staffing need, pulls ranked candidates from the bench, and shows you why each one ranked. Everything below loaded in about 3 seconds against the live system.</div>
|
||||
<div id="ch4-demo"><div class="loading">Running demo query…</div></div>
|
||||
</div>
|
||||
|
||||
<div class="chapter">
|
||||
<div class="num">Chapter 5</div>
|
||||
<h2>Every action compounds — the CRM-killer</h2>
|
||||
<div class="lede">A CRM stores. This system compounds. Every successful fill, every no-show, every phone call becomes a re-ranking signal on the next query. Below is the live playbook memory state. The number grows as the app gets used.</div>
|
||||
<div id="ch5-memory"><div class="loading">Reading playbook memory…</div></div>
|
||||
</div>
|
||||
|
||||
<div class="chapter">
|
||||
<div class="num">Chapter 6</div>
|
||||
<h2>Try it yourself</h2>
|
||||
<div class="lede">Type any staffing question. The system picks the right search path (smart-parse, semantic discovery, analytics), shows what it understood, and returns ranked results with memory signal.</div>
|
||||
<div class="try-box">
|
||||
<input type="text" id="try-q" placeholder="e.g. reliable forklift operators in Chicago with OSHA certs" onkeydown="if(event.key==='Enter')runTry()">
|
||||
<button id="try-btn" onclick="runTry()">Ask</button>
|
||||
<div id="try-out" style="margin-top:16px"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<div class="footer">Lakehouse · demo instance · <a href="proof">architecture & benchmarks</a></div>
|
||||
|
||||
<script>
|
||||
var P=location.pathname.indexOf('/lakehouse')>=0?'/lakehouse':'';
|
||||
var A=location.origin+P;
|
||||
|
||||
// DOM helpers — all dynamic content goes through these. No innerHTML
|
||||
// anywhere in the script; every API-derived string passes through
|
||||
// textContent so no injection path regardless of upstream data.
|
||||
function el(tag, cls, text){
|
||||
var e=document.createElement(tag);
|
||||
if(cls) e.className=cls;
|
||||
if(text!==undefined && text!==null) e.textContent=String(text);
|
||||
return e;
|
||||
}
|
||||
function setAttrs(e, obj){
|
||||
for(var k in obj) if(Object.prototype.hasOwnProperty.call(obj,k)) e.setAttribute(k,obj[k]);
|
||||
return e;
|
||||
}
|
||||
|
||||
function api(path, body){
|
||||
return fetch(A+path,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(body||{})}).then(function(r){return r.json()});
|
||||
}
|
||||
function apiGet(path){
|
||||
return fetch(A+path).then(function(r){return r.json()});
|
||||
}
|
||||
|
||||
window.addEventListener('load',function(){
|
||||
loadChapter1();
|
||||
loadChapter2();
|
||||
loadChapter3();
|
||||
loadChapter4();
|
||||
loadChapter5();
|
||||
});
|
||||
|
||||
// ─── Chapter 1 ────────────────────────────────────────────
|
||||
function loadChapter1(){
|
||||
Promise.all([
|
||||
api('/intelligence/staffing_forecast',{}).catch(function(){return {}}),
|
||||
apiGet('/api/vectors/playbook_memory/stats').catch(function(){return {}}),
|
||||
apiGet('/api/catalog/datasets').catch(function(){return []}),
|
||||
]).then(function(all){
|
||||
var f=all[0]||{}, mem=all[1]||{}, ds=all[2]||[];
|
||||
var totalRows=0;
|
||||
if(Array.isArray(ds)) ds.forEach(function(d){totalRows+=(d.row_count||0)});
|
||||
var host=document.getElementById('ch1-stats');host.textContent='';
|
||||
addStat(host,'$'+(f.total_cost||0).toLocaleString('en-US',{maximumFractionDigits:0}),'Construction pipeline',(f.permit_count||0)+' permits last 30 days','accent-b');
|
||||
addStat(host,(f.total_estimated_workers||0).toLocaleString(),'Predicted worker demand','split across '+((f.forecast||[]).length)+' roles','accent-w');
|
||||
addStat(host,(totalRows||0).toLocaleString(),'Rows under management',(Array.isArray(ds)?ds.length:0)+' datasets · real + synthetic','accent-a');
|
||||
addStat(host,(mem.entries||0).toLocaleString(),'Playbooks remembered',(mem.total_names_endorsed||0)+' endorsed worker-tags','accent-l');
|
||||
|
||||
var narrEl=document.getElementById('ch1-narr');
|
||||
narrEl.textContent='';
|
||||
var critical=(f.critical_roles||0), tight=(f.tight_roles||0);
|
||||
if(critical>0){
|
||||
narrEl.appendChild(document.createTextNode('The system has already flagged '));
|
||||
narrEl.appendChild(el('strong',null,critical+' role'+(critical!==1?'s':'')+' at critical coverage'));
|
||||
narrEl.appendChild(document.createTextNode(' — supply is below demand. A CRM would surface this only if someone ran the report. This system surfaces it before you log in.'));
|
||||
} else if(tight>0){
|
||||
narrEl.appendChild(document.createTextNode('The system flagged '));
|
||||
narrEl.appendChild(el('strong',null,tight+' role'+(tight!==1?'s':'')+' as tight'));
|
||||
narrEl.appendChild(document.createTextNode(' — watch for supply compression. Pre-emptive signal, no query required.'));
|
||||
} else {
|
||||
narrEl.textContent='No coverage gaps right now — bench exceeds predicted demand across all roles. The system is tracking this continuously; check back tomorrow and the numbers will have moved.';
|
||||
}
|
||||
}).catch(function(e){
|
||||
var h=document.getElementById('ch1-stats');h.textContent='';h.appendChild(el('div','err','Live state unavailable: '+(e.message||e)));
|
||||
});
|
||||
document.getElementById('hdr-time').textContent='Snapshot · '+new Date().toLocaleTimeString();
|
||||
}
|
||||
function addStat(host,n,l,sub,cls){
|
||||
var d=el('div','card stat-lg '+(cls||''));
|
||||
d.appendChild(el('div','n',n));
|
||||
d.appendChild(el('div','l',l));
|
||||
d.appendChild(el('div','sub',sub||''));
|
||||
host.appendChild(d);
|
||||
}
|
||||
|
||||
// ─── Chapter 2 ────────────────────────────────────────────
|
||||
function loadChapter2(){
|
||||
api('/intelligence/market',{}).then(function(r){
|
||||
var host=document.getElementById('ch2-permits');host.textContent='';
|
||||
if(r.error){host.appendChild(el('div','err','Permit feed error: '+r.error));return}
|
||||
var byType=r.by_type||[];
|
||||
if(byType.length===0){host.appendChild(el('div','narr','No permits returned by the Chicago API right now.'));return}
|
||||
byType.slice(0,8).forEach(function(t){
|
||||
var det=document.createElement('details');
|
||||
var summary=document.createElement('summary');
|
||||
summary.appendChild(el('span','pill real','LIVE · DATA.CITYOFCHICAGO.ORG'));
|
||||
summary.appendChild(document.createTextNode(' '));
|
||||
var wt=el('strong',null,t.work_type||'General construction');
|
||||
summary.appendChild(wt);
|
||||
summary.appendChild(document.createTextNode(' · '+(t.permits||0)+' permits · $'+(t.total_cost||0).toLocaleString('en-US',{maximumFractionDigits:0})+' · est '+(t.estimated_workers||0)+' workers needed'));
|
||||
det.appendChild(summary);
|
||||
var body=el('div','body');
|
||||
body.appendChild(document.createTextNode('Likely roles: '+((t.needed_roles||[]).join(', ')||'—')+'. '));
|
||||
body.appendChild(document.createTextNode('At industry heuristic (~$150K per worker), this work type alone implies '));
|
||||
body.appendChild(el('strong',null,(t.estimated_workers||0)+' staffing opportunities'));
|
||||
body.appendChild(document.createTextNode(' over the next 30-60 days.'));
|
||||
det.appendChild(body);
|
||||
host.appendChild(det);
|
||||
});
|
||||
}).catch(function(e){
|
||||
var h=document.getElementById('ch2-permits');h.textContent='';h.appendChild(el('div','err',e.message||String(e)));
|
||||
});
|
||||
}
|
||||
|
||||
// ─── Chapter 3 ────────────────────────────────────────────
|
||||
function loadChapter3(){
|
||||
apiGet('/api/catalog/datasets').then(function(ds){
|
||||
var host=document.getElementById('ch3-datasets');host.textContent='';
|
||||
if(!Array.isArray(ds)){host.appendChild(el('div','err','No datasets returned.'));return}
|
||||
var bigOnes=ds.filter(function(d){return (d.row_count||0)>=500}).sort(function(a,b){return (b.row_count||0)-(a.row_count||0)});
|
||||
var playbookKeys=['successful_playbooks','playbook_memory','kb_'];
|
||||
bigOnes.slice(0,12).forEach(function(d){
|
||||
var row=el('div','row');
|
||||
var left=document.createElement('div');left.style.flex='1';left.style.minWidth='0';
|
||||
var t=el('div','title');
|
||||
var pillCls='synth', pillLabel='SWAP FOR YOUR DATA';
|
||||
if(playbookKeys.some(function(k){return (d.name||'').indexOf(k)>=0})){pillCls='flow';pillLabel='SYSTEM-GENERATED'}
|
||||
else if((d.name||'').indexOf('threat_')===0){pillCls='flow';pillLabel='SYSTEM-GENERATED'}
|
||||
t.appendChild(el('span','pill '+pillCls,pillLabel));
|
||||
t.appendChild(document.createTextNode(' '+(d.name||'(unnamed)')));
|
||||
var m=el('div','meta',(d.columns?d.columns.length+' columns · ':'')+(d.objects?d.objects.length+' parquet file'+(d.objects.length!==1?'s':''):''));
|
||||
left.appendChild(t);left.appendChild(m);
|
||||
var v=el('div','val',(d.row_count||0).toLocaleString()+' rows');
|
||||
row.appendChild(left);row.appendChild(v);host.appendChild(row);
|
||||
});
|
||||
}).catch(function(e){
|
||||
var h=document.getElementById('ch3-datasets');h.textContent='';h.appendChild(el('div','err','Catalog unavailable: '+(e.message||e)));
|
||||
});
|
||||
}
|
||||
|
||||
// ─── Chapter 4 ────────────────────────────────────────────
|
||||
function loadChapter4(){
|
||||
api('/intelligence/permit_contracts',{}).then(function(r){
|
||||
var host=document.getElementById('ch4-demo');host.textContent='';
|
||||
if(!r.contracts||r.contracts.length===0){host.appendChild(el('div','loading','No contracts returned.'));return}
|
||||
var c=r.contracts[0];
|
||||
var p=c.permit||{}, prop=c.proposed||{}, tl=c.timeline||{};
|
||||
var card=el('div','card accent-b');
|
||||
var title=el('div',null,'$'+(p.cost||0).toLocaleString()+' · '+(p.work_type||''));
|
||||
title.style.cssText='font-size:15px;font-weight:600;color:#e6edf3';
|
||||
card.appendChild(title);
|
||||
var addr=el('div',null,(p.address||'')+' · Chicago, IL · filed '+(p.issue_date||''));
|
||||
addr.style.cssText='color:#8b949e;font-size:12px;margin-top:2px';
|
||||
card.appendChild(addr);
|
||||
|
||||
card.appendChild(el('div','step-label','STEP 1 · Derive staffing need'));
|
||||
var s1=el('div','step-body');
|
||||
s1.appendChild(document.createTextNode('Industry heuristic: ~1 worker per $150K of permit cost, capped 2-8. Resulting contract: '));
|
||||
s1.appendChild(el('strong',null,(prop.count||0)+'× '+(prop.role||'')));
|
||||
s1.appendChild(document.createTextNode(' in Chicago, IL.'));
|
||||
card.appendChild(s1);
|
||||
|
||||
card.appendChild(el('div','step-label','STEP 2 · Narrow via SQL ('+((prop.pool_size||0).toLocaleString())+' candidates match)'));
|
||||
card.appendChild(el('div','step-body',"role = '"+(prop.role||'')+"' AND state = 'IL' AND city = 'Chicago' AND CAST(availability AS DOUBLE) > 0.5"));
|
||||
|
||||
card.appendChild(el('div','step-label','STEP 3 · Rank via semantic match + playbook boost'));
|
||||
host.appendChild(card);
|
||||
|
||||
var list=document.createElement('div');list.style.marginTop='6px';
|
||||
(prop.candidates||[]).slice(0,5).forEach(function(cand,i){
|
||||
var w=el('div','worker');
|
||||
var initials=(cand.name||'?').split(' ').map(function(s){return (s[0]||'').toUpperCase()}).join('').substring(0,2);
|
||||
w.appendChild(el('div','av',initials));
|
||||
var info=el('div','info');
|
||||
var nm=el('div','nm',cand.name||cand.doc_id||'?');
|
||||
if((cand.playbook_boost||0)>0){
|
||||
var ncit=(cand.playbook_citations||[]).length;
|
||||
nm.appendChild(el('span','boost-chip','Endorsed · '+ncit+' past fill'+(ncit!==1?'s':'')));
|
||||
}
|
||||
info.appendChild(nm);
|
||||
var why=cand.doc_id+' · '+(cand.playbook_boost>0?'boosted +'+cand.playbook_boost.toFixed(3)+' by memory · ':'')+'semantic score '+(cand.score||0).toFixed(3);
|
||||
info.appendChild(el('div','why',why));
|
||||
w.appendChild(info);
|
||||
w.appendChild(el('div','score','#'+(i+1)));
|
||||
list.appendChild(w);
|
||||
});
|
||||
card.appendChild(list);
|
||||
|
||||
if(c.discovered_pattern||c.pattern_matched>0){
|
||||
var mem=el('div','mem-chip');
|
||||
mem.appendChild(el('span','l','MEMORY ('+(c.pattern_matched||0)+' past playbooks):'));
|
||||
mem.appendChild(document.createTextNode(' '+(c.discovered_pattern||'memory is sparse for this role+geo')));
|
||||
card.appendChild(mem);
|
||||
}
|
||||
|
||||
if(tl.days_to_deadline!==undefined){
|
||||
var tm=document.createElement('div');
|
||||
tm.style.cssText='margin-top:12px;padding-top:10px;border-top:1px solid #171d27;font-size:12px;color:#8b949e';
|
||||
tm.appendChild(el('strong',null,'Timeline. '));
|
||||
tm.childNodes[0].style && (tm.childNodes[0].style.color='#c9d1d9');
|
||||
var dd=tl.days_to_deadline;
|
||||
var when=(dd<=0?Math.abs(dd)+' days overdue':dd+' days out');
|
||||
tm.appendChild(document.createTextNode('Staffing window opens '+(tl.staffing_window_opens||'')+' ('+when+'). Construction starts '+(tl.estimated_construction_start||'')+'. Urgency: '));
|
||||
tm.appendChild(el('strong',null,(tl.urgency||'').toUpperCase()));
|
||||
tm.appendChild(document.createTextNode('.'));
|
||||
card.appendChild(tm);
|
||||
}
|
||||
|
||||
var note=el('div','narr');
|
||||
note.appendChild(el('strong',null,'This is the CRM gap. '));
|
||||
note.appendChild(document.createTextNode('A CRM would tell you the permit exists. It would NOT narrow to matching candidates, would NOT boost past-successful workers, would NOT flag patterns across similar past fills, would NOT count down to the deadline. All that happens here, pre-computed, before the staffer clicks anything.'));
|
||||
host.appendChild(note);
|
||||
}).catch(function(e){
|
||||
var h=document.getElementById('ch4-demo');h.textContent='';h.appendChild(el('div','err','Demo failed: '+(e.message||e)));
|
||||
});
|
||||
}
|
||||
|
||||
// ─── Chapter 5 ────────────────────────────────────────────
|
||||
function loadChapter5(){
|
||||
apiGet('/api/vectors/playbook_memory/stats').then(function(mem){
|
||||
var host=document.getElementById('ch5-memory');host.textContent='';
|
||||
var card=el('div','card accent-l');
|
||||
var big=el('div',null,(mem.entries||0).toLocaleString());
|
||||
big.style.cssText='font-size:30px;font-weight:800;color:#e6edf3;line-height:1';
|
||||
card.appendChild(big);
|
||||
var lb=el('div',null,'Playbooks in memory');
|
||||
lb.style.cssText='font-size:10px;color:#545d68;text-transform:uppercase;letter-spacing:1.2px;margin-top:8px;font-weight:600';
|
||||
card.appendChild(lb);
|
||||
var sub=el('div',null,(mem.total_names_endorsed||0).toLocaleString()+' endorsed worker-tags · each one boosts future matching queries');
|
||||
sub.style.cssText='font-size:12px;color:#8b949e;margin-top:4px';
|
||||
card.appendChild(sub);
|
||||
host.appendChild(card);
|
||||
|
||||
if(mem.sample&&mem.sample.length){
|
||||
var hdr=document.createElement('div');
|
||||
hdr.style.cssText='color:#545d68;font-size:11px;text-transform:uppercase;letter-spacing:1.4px;font-weight:600;margin:14px 0 8px';
|
||||
hdr.textContent='Sample playbooks held in memory right now';
|
||||
host.appendChild(hdr);
|
||||
mem.sample.slice(0,5).forEach(function(pb){
|
||||
var row=el('div','row');
|
||||
var left=document.createElement('div');left.style.flex='1';left.style.minWidth='0';
|
||||
left.appendChild(el('div','title',pb.operation||'(no operation)'));
|
||||
left.appendChild(el('div','meta',(pb.city||'')+', '+(pb.state||'')+' · endorsed: '+((pb.endorsed||[]).slice(0,3).join(', ')||'—')));
|
||||
row.appendChild(left);host.appendChild(row);
|
||||
});
|
||||
}
|
||||
|
||||
var narr=el('div','narr');
|
||||
narr.appendChild(el('strong',null,'What this means for a staffer. '));
|
||||
narr.appendChild(document.createTextNode('When you log a successful fill, this counter grows by one. Next time someone searches for a similar role+city, the system finds your past pick and boosts them to the top. Over months, the system becomes a record of who has actually worked out — not just who is on the roster.'));
|
||||
host.appendChild(narr);
|
||||
}).catch(function(e){
|
||||
var h=document.getElementById('ch5-memory');h.textContent='';h.appendChild(el('div','err','Memory unavailable: '+(e.message||e)));
|
||||
});
|
||||
}
|
||||
|
||||
// ─── Chapter 6 ────────────────────────────────────────────
|
||||
function runTry(){
|
||||
var q=document.getElementById('try-q').value.trim();if(!q)return;
|
||||
var btn=document.getElementById('try-btn'),out=document.getElementById('try-out');
|
||||
btn.disabled=true;btn.textContent='Thinking…';out.textContent='';out.appendChild(el('div','loading','Running…'));
|
||||
api('/intelligence/chat',{message:q}).then(function(d){
|
||||
btn.disabled=false;btn.textContent='Ask';out.textContent='';
|
||||
var card=el('div','card accent-b');
|
||||
if(d.understood&&d.understood.length){
|
||||
var tags=document.createElement('div');tags.style.cssText='display:flex;gap:6px;flex-wrap:wrap;margin-bottom:10px';
|
||||
d.understood.forEach(function(u){
|
||||
var tg=el('span',null,u);
|
||||
tg.style.cssText='padding:3px 10px;border-radius:10px;font-size:11px;background:#1a274420;color:#58a6ff;border:1px solid #1a274480';
|
||||
tags.appendChild(tg);
|
||||
});
|
||||
card.appendChild(tags);
|
||||
}
|
||||
var sum=el('div',null,d.summary||'(no summary)');
|
||||
sum.style.cssText='color:#c9d1d9;font-size:13px;margin-bottom:10px';
|
||||
card.appendChild(sum);
|
||||
|
||||
if(d.pattern_playbooks_matched>0&&d.discovered_pattern){
|
||||
var mem=el('div','mem-chip');
|
||||
mem.appendChild(el('span','l','MEMORY ('+d.pattern_playbooks_matched+' playbooks):'));
|
||||
mem.appendChild(document.createTextNode(' '+d.discovered_pattern));
|
||||
card.appendChild(mem);
|
||||
}
|
||||
|
||||
var workers=d.sql_results||d.vector_results||d.results||[];
|
||||
workers.slice(0,5).forEach(function(w,i){
|
||||
var row=el('div','worker');
|
||||
var nm=w.name||(w.text||'').split('—')[0].trim()||w.doc_id||'?';
|
||||
var initials=nm.split(' ').map(function(s){return (s[0]||'').toUpperCase()}).join('').substring(0,2);
|
||||
row.appendChild(el('div','av',initials));
|
||||
var info=el('div','info');
|
||||
var n=el('div','nm',nm);
|
||||
if((w.playbook_boost||0)>0){
|
||||
n.appendChild(el('span','boost-chip','Endorsed · '+((w.playbook_citations||[]).length||'?')+' past fill(s)'));
|
||||
}
|
||||
info.appendChild(n);
|
||||
var bits=[];
|
||||
if(w.role) bits.push(w.role);
|
||||
if(w.city&&w.state) bits.push(w.city+', '+w.state);
|
||||
if(w.rel!==undefined) bits.push('reliability '+Math.round(w.rel*100)+'%');
|
||||
if(w.avail!==undefined) bits.push('availability '+Math.round(w.avail*100)+'%');
|
||||
info.appendChild(el('div','why',bits.join(' · ')||'AI semantic match'));
|
||||
row.appendChild(info);
|
||||
row.appendChild(el('div','score','#'+(i+1)));
|
||||
card.appendChild(row);
|
||||
});
|
||||
var meta=el('div',null,(d.duration_ms||0)+'ms · type='+(d.type||'smart_search'));
|
||||
meta.style.cssText='color:#545d68;font-size:11px;margin-top:10px';
|
||||
card.appendChild(meta);
|
||||
out.appendChild(card);
|
||||
}).catch(function(e){
|
||||
btn.disabled=false;btn.textContent='Ask';
|
||||
out.textContent='';out.appendChild(el('div','err',e.message||String(e)));
|
||||
});
|
||||
}
|
||||
</script>
|
||||
</body></html>
|
||||
@@ -0,0 +1,178 @@
|
||||
// Phase 45 slice 2 — context7 HTTP bridge.
|
||||
//
|
||||
// Exposes an HTTP surface the Rust gateway consumes to check external
|
||||
// doc drift on playbooks. Wraps context7's public API:
|
||||
//
|
||||
// https://context7.com/api/v1/search?query=<name> → resolve
|
||||
// https://context7.com/api/v1/<lib-id>/cli?type=txt&tokens=N → docs
|
||||
//
|
||||
// Own port so a failure here never tips over the mcp-server on :3700.
|
||||
// Cache is in-memory (5 min TTL) — context7 rate-limits by IP, and
|
||||
// gateway drift-checks are the hot caller.
|
||||
//
|
||||
// Endpoints:
|
||||
// GET /health health + cache stats
|
||||
// GET /docs/:tool resolve + fetch + return descriptor
|
||||
// GET /docs/:tool/diff?since=X return drift vs content hash X
|
||||
// GET /cache dump current cache (debugging)
|
||||
|
||||
import { createHash } from "node:crypto";
|
||||
|
||||
const PORT = Number(process.env.CONTEXT7_BRIDGE_PORT ?? 3900);
|
||||
const CONTEXT7_BASE = (process.env.CONTEXT7_BASE_URL ?? "https://context7.com/api/v1").replace(/\/+$/, "");
|
||||
const TOKENS_PER_FETCH = 1500; // enough for a drift-meaningful slice, not so much we hammer
|
||||
const CACHE_TTL_MS = 5 * 60 * 1000;
|
||||
|
||||
interface CachedDoc {
|
||||
tool: string;
|
||||
library_id: string; // context7 canonical, e.g. "/docker/docs"
|
||||
title: string;
|
||||
last_updated: string | null;
|
||||
snippet_hash: string; // SHA-256 of first TOKENS_PER_FETCH of fetched docs
|
||||
docs_preview: string; // first 400 chars, returned in responses for display
|
||||
retrieved_at: string; // ISO when WE last fetched
|
||||
source_url: string; // human-facing context7 URL
|
||||
}
|
||||
|
||||
const cache = new Map<string, { entry: CachedDoc; at: number }>();
|
||||
|
||||
function normalizeTool(s: string): string {
|
||||
return s.trim().toLowerCase();
|
||||
}
|
||||
|
||||
async function resolveLibraryId(tool: string): Promise<{ id: string; title: string; last_updated: string | null } | null> {
|
||||
const r = await fetch(`${CONTEXT7_BASE}/search?query=${encodeURIComponent(tool)}`, {
|
||||
signal: AbortSignal.timeout(10000),
|
||||
});
|
||||
if (!r.ok) throw new Error(`context7 search ${r.status}: ${await r.text().catch(() => "?")}`);
|
||||
const j = await r.json() as any;
|
||||
const results = Array.isArray(j.results) ? j.results : [];
|
||||
// First result is top-ranked; context7 sorts by its internal score.
|
||||
// Skip "state != finalized" to avoid unstable previews.
|
||||
const pick = results.find((x: any) => x?.state === "finalized") ?? results[0];
|
||||
if (!pick?.id) return null;
|
||||
return {
|
||||
id: String(pick.id),
|
||||
title: String(pick.title ?? tool),
|
||||
last_updated: typeof pick.lastUpdateDate === "string" ? pick.lastUpdateDate : null,
|
||||
};
|
||||
}
|
||||
|
||||
async function fetchDocsText(libraryId: string): Promise<string> {
|
||||
// Strip the leading slash on library_id so the URL doesn't have a
|
||||
// double slash. context7 returns ids like "/docker/docs" but the
|
||||
// fetch path wants "docker/docs".
|
||||
const cleanId = libraryId.replace(/^\/+/, "");
|
||||
const url = `${CONTEXT7_BASE}/${cleanId}?type=txt&tokens=${TOKENS_PER_FETCH}`;
|
||||
const r = await fetch(url, { signal: AbortSignal.timeout(20000) });
|
||||
if (!r.ok) throw new Error(`context7 fetch ${r.status} on ${cleanId}: ${await r.text().catch(() => "?")}`);
|
||||
return await r.text();
|
||||
}
|
||||
|
||||
function hashContent(s: string): string {
|
||||
return createHash("sha256").update(s).digest("hex").slice(0, 16);
|
||||
}
|
||||
|
||||
async function getCurrent(tool: string): Promise<CachedDoc> {
|
||||
const key = normalizeTool(tool);
|
||||
const cached = cache.get(key);
|
||||
if (cached && Date.now() - cached.at < CACHE_TTL_MS) {
|
||||
return cached.entry;
|
||||
}
|
||||
const resolved = await resolveLibraryId(key);
|
||||
if (!resolved) {
|
||||
throw new Error(`no context7 library found for '${tool}'`);
|
||||
}
|
||||
const text = await fetchDocsText(resolved.id);
|
||||
const entry: CachedDoc = {
|
||||
tool: key,
|
||||
library_id: resolved.id,
|
||||
title: resolved.title,
|
||||
last_updated: resolved.last_updated,
|
||||
snippet_hash: hashContent(text),
|
||||
docs_preview: text.slice(0, 400),
|
||||
retrieved_at: new Date().toISOString(),
|
||||
source_url: `https://context7.com${resolved.id.startsWith("/") ? resolved.id : "/" + resolved.id}`,
|
||||
};
|
||||
cache.set(key, { entry, at: Date.now() });
|
||||
return entry;
|
||||
}
|
||||
|
||||
function jsonResponse(body: unknown, status: number = 200): Response {
|
||||
return new Response(JSON.stringify(body), {
|
||||
status,
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
}
|
||||
|
||||
Bun.serve({
|
||||
port: PORT,
|
||||
hostname: "0.0.0.0",
|
||||
async fetch(req) {
|
||||
const url = new URL(req.url);
|
||||
|
||||
if (url.pathname === "/health") {
|
||||
return jsonResponse({
|
||||
status: "ok",
|
||||
cache_size: cache.size,
|
||||
context7_base: CONTEXT7_BASE,
|
||||
ttl_ms: CACHE_TTL_MS,
|
||||
});
|
||||
}
|
||||
|
||||
if (url.pathname === "/cache") {
|
||||
const dump = Array.from(cache.entries()).map(([k, v]) => ({
|
||||
tool: k,
|
||||
library_id: v.entry.library_id,
|
||||
snippet_hash: v.entry.snippet_hash,
|
||||
retrieved_at: v.entry.retrieved_at,
|
||||
age_ms: Date.now() - v.at,
|
||||
}));
|
||||
return jsonResponse({ entries: dump });
|
||||
}
|
||||
|
||||
// GET /docs/:tool — return current descriptor (fetches + caches)
|
||||
const m1 = url.pathname.match(/^\/docs\/([^/]+)$/);
|
||||
if (m1 && req.method === "GET") {
|
||||
const tool = decodeURIComponent(m1[1]);
|
||||
try {
|
||||
const entry = await getCurrent(tool);
|
||||
return jsonResponse(entry);
|
||||
} catch (e) {
|
||||
return jsonResponse({ error: (e as Error).message }, 404);
|
||||
}
|
||||
}
|
||||
|
||||
// GET /docs/:tool/diff?since=hash — compare current vs recorded hash
|
||||
const m2 = url.pathname.match(/^\/docs\/([^/]+)\/diff$/);
|
||||
if (m2 && req.method === "GET") {
|
||||
const tool = decodeURIComponent(m2[1]);
|
||||
const since = url.searchParams.get("since");
|
||||
if (!since) {
|
||||
return jsonResponse({ error: "query param 'since' (snippet_hash) required" }, 400);
|
||||
}
|
||||
try {
|
||||
const entry = await getCurrent(tool);
|
||||
const drifted = entry.snippet_hash !== since;
|
||||
return jsonResponse({
|
||||
tool: entry.tool,
|
||||
drifted,
|
||||
previous_snippet_hash: since,
|
||||
current_snippet_hash: entry.snippet_hash,
|
||||
library_id: entry.library_id,
|
||||
title: entry.title,
|
||||
last_updated_upstream: entry.last_updated,
|
||||
retrieved_at: entry.retrieved_at,
|
||||
source_url: entry.source_url,
|
||||
docs_preview: drifted ? entry.docs_preview : null,
|
||||
});
|
||||
} catch (e) {
|
||||
return jsonResponse({ error: (e as Error).message }, 404);
|
||||
}
|
||||
}
|
||||
|
||||
return jsonResponse({ error: "not found", paths: ["/health", "/docs/:tool", "/docs/:tool/diff?since=HASH", "/cache"] }, 404);
|
||||
},
|
||||
});
|
||||
|
||||
console.log(`[context7-bridge] listening on :${PORT} (context7 base: ${CONTEXT7_BASE})`);
|
||||
@@ -0,0 +1,355 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Lakehouse — Staffing Search</title>
|
||||
<style>
|
||||
*{margin:0;padding:0;box-sizing:border-box}
|
||||
body{font-family:'Inter','SF Pro',system-ui,sans-serif;background:#0a0a0f;color:#d4d4d8;line-height:1.6}
|
||||
.hero{background:linear-gradient(135deg,#0f172a 0%,#1e1b4b 50%,#0f172a 100%);padding:30px;border-bottom:1px solid #1e293b;text-align:center}
|
||||
.hero h1{font-size:24px;font-weight:700;background:linear-gradient(to right,#f472b6,#818cf8);-webkit-background-clip:text;-webkit-text-fill-color:transparent;margin-bottom:6px}
|
||||
.hero p{color:#94a3b8;font-size:13px}
|
||||
.container{max-width:900px;margin:0 auto;padding:24px 16px}
|
||||
.search-box{background:#111827;border:2px solid #334155;border-radius:12px;padding:20px;margin-bottom:24px;transition:border-color .2s}
|
||||
.search-box:focus-within{border-color:#818cf8}
|
||||
.search-row{display:flex;gap:10px}
|
||||
.search-input{flex:1;background:#0a0a0f;border:1px solid #1e293b;border-radius:8px;padding:14px 18px;color:#e2e8f0;font-size:15px;outline:none}
|
||||
.search-input::placeholder{color:#475569}
|
||||
.search-btn{background:#7c3aed;border:none;color:#fff;padding:14px 28px;border-radius:8px;cursor:pointer;font-size:14px;font-weight:600;white-space:nowrap}
|
||||
.search-btn:hover{background:#6d28d9}
|
||||
.search-btn:disabled{background:#334155;cursor:wait}
|
||||
.filters{display:flex;gap:8px;margin-top:12px;flex-wrap:wrap}
|
||||
.filter{background:#1e293b;border:1px solid #334155;border-radius:6px;padding:6px 12px;font-size:12px;color:#94a3b8;outline:none}
|
||||
.filter select,.filter input{background:transparent;border:none;color:#e2e8f0;font-size:12px;outline:none;width:auto}
|
||||
.filter select{cursor:pointer}
|
||||
.filter label{color:#64748b;margin-right:4px}
|
||||
.examples{display:flex;gap:8px;margin-top:12px;flex-wrap:wrap}
|
||||
.example{background:#0d1117;border:1px solid #1e293b;border-radius:20px;padding:4px 14px;font-size:11px;color:#818cf8;cursor:pointer;transition:all .15s}
|
||||
.example:hover{background:#1e1b4b;border-color:#818cf8}
|
||||
.results{margin-top:8px}
|
||||
.result-header{display:flex;justify-content:space-between;align-items:center;margin-bottom:16px;flex-wrap:wrap;gap:8px}
|
||||
.result-header .count{font-size:14px;color:#e2e8f0;font-weight:600}
|
||||
.result-header .meta{font-size:11px;color:#64748b}
|
||||
.badge{display:inline-block;padding:2px 10px;border-radius:20px;font-size:10px;font-weight:600}
|
||||
.badge.green{background:#052e16;color:#34d399;border:1px solid #166534}
|
||||
.badge.purple{background:#1e1047;color:#a78bfa;border:1px solid #5b21b6}
|
||||
.badge.blue{background:#0c1a3d;color:#60a5fa;border:1px solid #1e40af}
|
||||
.worker{background:#111827;border:1px solid #1e293b;border-radius:10px;padding:18px;margin-bottom:12px;transition:border-color .2s}
|
||||
.worker:hover{border-color:#334155}
|
||||
.worker .top{display:flex;justify-content:space-between;align-items:flex-start;gap:12px}
|
||||
.worker .name{font-size:16px;font-weight:700;color:#e2e8f0}
|
||||
.worker .role{font-size:13px;color:#818cf8;margin-top:2px}
|
||||
.worker .location{font-size:12px;color:#64748b;margin-top:2px}
|
||||
.worker .score{text-align:right}
|
||||
.worker .score .num{font-size:24px;font-weight:800;color:#34d399}
|
||||
.worker .score .label{font-size:10px;color:#64748b;text-transform:uppercase}
|
||||
.worker .details{display:flex;gap:16px;margin-top:12px;flex-wrap:wrap}
|
||||
.worker .detail{font-size:11px}
|
||||
.worker .detail .dt{color:#64748b;text-transform:uppercase;font-size:10px;letter-spacing:0.5px}
|
||||
.worker .detail .dd{color:#d4d4d8;margin-top:2px}
|
||||
.worker .verified{display:inline-flex;align-items:center;gap:4px;font-size:10px;color:#34d399;margin-top:8px}
|
||||
.empty{text-align:center;color:#475569;padding:40px;font-size:14px}
|
||||
.loading{text-align:center;padding:30px;color:#818cf8}
|
||||
.mode-toggle{display:flex;gap:4px;background:#0d1117;border-radius:8px;padding:3px;margin-bottom:16px}
|
||||
.mode-btn{padding:8px 16px;border-radius:6px;border:none;cursor:pointer;font-size:12px;font-weight:500;color:#94a3b8;background:transparent}
|
||||
.mode-btn.active{background:#7c3aed;color:#fff}
|
||||
.footer{text-align:center;padding:20px;color:#475569;font-size:11px;border-top:1px solid #1e293b;margin-top:30px}
|
||||
.footer a{color:#818cf8;text-decoration:none}
|
||||
@media(max-width:768px){
|
||||
.search-row{flex-direction:column}
|
||||
.worker .top{flex-direction:column}
|
||||
.worker .score{text-align:left}
|
||||
.worker .details{flex-direction:column;gap:8px}
|
||||
.filters{flex-direction:column}
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="hero">
|
||||
<h1>Search 500,000 Workers</h1>
|
||||
<p>Type what you need in plain English — the AI understands meaning, not just keywords</p>
|
||||
</div>
|
||||
<div class="container">
|
||||
<div class="search-box">
|
||||
<div class="search-row">
|
||||
<input class="search-input" id="query" type="text" placeholder="e.g. reliable forklift operator for warehouse in Illinois" autofocus>
|
||||
<button class="search-btn" id="search-btn" onclick="doSearch()">Search</button>
|
||||
</div>
|
||||
<div class="filters">
|
||||
<div class="filter"><label>State:</label><select id="f-state"><option value="">Any</option><option>IL</option><option>IN</option><option>OH</option><option>MO</option><option>TN</option><option>KY</option><option>WI</option><option>MI</option></select></div>
|
||||
<div class="filter"><label>Role:</label><select id="f-role"><option value="">Any</option><option>Forklift Operator</option><option>Machine Operator</option><option>Assembler</option><option>Loader</option><option>Quality Tech</option><option>Welder</option><option>Sanitation Worker</option><option>Shipping Clerk</option><option>Production Worker</option><option>Maintenance Tech</option></select></div>
|
||||
<div class="filter"><label>Min Reliability:</label><input id="f-rel" type="number" min="0" max="1" step="0.1" value="0.5" style="width:50px"></div>
|
||||
</div>
|
||||
<div class="examples">
|
||||
<span class="example" onclick="tryExample(this)">warehouse help in Illinois</span>
|
||||
<span class="example" onclick="tryExample(this)">dependable machine operator with CNC</span>
|
||||
<span class="example" onclick="tryExample(this)">safety trained for chemical plant</span>
|
||||
<span class="example" onclick="tryExample(this)">bilingual shipping clerk</span>
|
||||
<span class="example" onclick="tryExample(this)">experienced welder Ohio</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="mode-toggle">
|
||||
<button class="mode-btn active" id="mode-ai" onclick="setMode('ai')">AI Search</button>
|
||||
<button class="mode-btn" id="mode-crm" onclick="setMode('crm')">CRM Keyword</button>
|
||||
<button class="mode-btn" id="mode-hybrid" onclick="setMode('hybrid')">Hybrid (best)</button>
|
||||
</div>
|
||||
|
||||
<div id="results"></div>
|
||||
|
||||
<div class="footer">
|
||||
<a href="proof">View Proof of Work</a> · Powered by Lakehouse · 500K workers · 673K AI-indexed chunks
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
window.onerror = function(msg, url, line) {
|
||||
document.body.insertAdjacentHTML('afterbegin',
|
||||
'<div style="background:#7f1d1d;color:#fca5a5;padding:12px;font-size:12px">JS Error: ' + msg + ' (line ' + line + ')</div>');
|
||||
};
|
||||
|
||||
const base = window.location.pathname.replace(/\/+$/, '');
|
||||
const GW = window.location.origin + base;
|
||||
let mode = 'ai';
|
||||
|
||||
function setMode(m) {
|
||||
mode = m;
|
||||
document.querySelectorAll('.mode-btn').forEach(b => b.classList.remove('active'));
|
||||
document.getElementById('mode-' + m).classList.add('active');
|
||||
}
|
||||
|
||||
function tryExample(el) {
|
||||
document.getElementById('query').value = el.textContent;
|
||||
doSearch();
|
||||
}
|
||||
|
||||
document.getElementById('query').addEventListener('keydown', function(e) {
|
||||
if (e.key === 'Enter') doSearch();
|
||||
});
|
||||
|
||||
async function doSearch() {
|
||||
const query = document.getElementById('query').value.trim();
|
||||
if (!query) return;
|
||||
|
||||
const state = document.getElementById('f-state').value;
|
||||
const role = document.getElementById('f-role').value;
|
||||
const rel = parseFloat(document.getElementById('f-rel').value) || 0.5;
|
||||
|
||||
const btn = document.getElementById('search-btn');
|
||||
btn.disabled = true;
|
||||
btn.textContent = 'Searching...';
|
||||
const searchLabel = effectiveMode === 'crm' ? 'by keyword' : effectiveMode === 'hybrid' ? 'with AI + filters' : 'with AI';
|
||||
document.getElementById('results').innerHTML = '<div class="loading">Searching ' + searchLabel + '...</div>';
|
||||
|
||||
const t0 = Date.now();
|
||||
|
||||
// ALWAYS use hybrid for AI modes — it handles filters AND works even
|
||||
// when HNSW isn't loaded (falls back to brute-force). Pure HNSW mode
|
||||
// hangs when the RAM index isn't built.
|
||||
const effectiveMode = (mode === 'crm') ? 'crm' : 'hybrid';
|
||||
|
||||
try {
|
||||
let data;
|
||||
|
||||
if (effectiveMode === 'crm') {
|
||||
// CRM keyword search — exact LIKE match
|
||||
let where = "resume_text LIKE '%" + query.replace(/'/g, "''") + "%'";
|
||||
if (state) where += " AND state = '" + state + "'";
|
||||
if (role) where += " AND role = '" + role + "'";
|
||||
where += " AND CAST(reliability AS DOUBLE) >= " + rel;
|
||||
const r = await fetch(GW + '/sql', {
|
||||
method: 'POST', headers: {'Content-Type': 'application/json'},
|
||||
body: JSON.stringify({sql: "SELECT worker_id, name, role, city, state, skills, certifications, archetype, ROUND(CAST(reliability AS DOUBLE),2) as reliability, ROUND(CAST(availability AS DOUBLE),2) as availability FROM workers_500k WHERE " + where + " ORDER BY CAST(reliability AS DOUBLE) DESC LIMIT 10"})
|
||||
});
|
||||
data = await r.json();
|
||||
renderCRMResults(data, query, Date.now() - t0);
|
||||
|
||||
} else {
|
||||
// Hybrid — SQL filters enforce structure, AI ranks by relevance
|
||||
// Always works — uses brute-force when HNSW isn't loaded
|
||||
let filter = "CAST(reliability AS DOUBLE) >= " + rel;
|
||||
if (state) filter += " AND state = '" + state + "'";
|
||||
if (role) filter += " AND role = '" + role + "'";
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(function() { controller.abort(); }, 15000);
|
||||
const r = await fetch(GW + '/search', {
|
||||
method: 'POST', headers: {'Content-Type': 'application/json'},
|
||||
body: JSON.stringify({
|
||||
question: query, index_name: 'workers_500k_v1',
|
||||
sql_filter: filter, dataset: 'workers_500k',
|
||||
id_column: 'worker_id', top_k: 10, generate: false
|
||||
}),
|
||||
signal: controller.signal
|
||||
});
|
||||
clearTimeout(timeout);
|
||||
data = await r.json();
|
||||
renderHybridResults(data, query, Date.now() - t0);
|
||||
}
|
||||
} catch (e) {
|
||||
document.getElementById('results').innerHTML = '<div class="empty">Search error: ' + e.message + '<br><br>If filters were set, try Hybrid mode. If the search is slow, the AI index may be loading — try again in 10 seconds.</div>';
|
||||
}
|
||||
|
||||
btn.disabled = false;
|
||||
btn.textContent = 'Search';
|
||||
}
|
||||
|
||||
function renderCRMResults(data, query, ms) {
|
||||
const el = document.getElementById('results');
|
||||
const rows = data.rows || [];
|
||||
el.innerHTML = '';
|
||||
|
||||
const header = document.createElement('div');
|
||||
header.className = 'result-header';
|
||||
header.innerHTML = '<span class="count">' + rows.length + ' results</span>' +
|
||||
'<span class="meta">CRM keyword match · "' + query + '" · ' + ms + 'ms</span>';
|
||||
el.appendChild(header);
|
||||
|
||||
if (!rows.length) {
|
||||
el.innerHTML += '<div class="empty">No exact keyword match found.<br>Try <b>AI Search</b> — it understands what you mean, not just the exact words.</div>';
|
||||
return;
|
||||
}
|
||||
rows.forEach(function(w) { el.appendChild(makeWorkerCard(w, null)); });
|
||||
}
|
||||
|
||||
function renderAIResults(data, query, ms) {
|
||||
const el = document.getElementById('results');
|
||||
const hits = data.results || [];
|
||||
el.innerHTML = '';
|
||||
|
||||
const header = document.createElement('div');
|
||||
header.className = 'result-header';
|
||||
header.innerHTML = '<span class="count">' + hits.length + ' results</span>' +
|
||||
'<span class="meta">AI vector search · understood meaning · ' + ms + 'ms</span>';
|
||||
el.appendChild(header);
|
||||
|
||||
if (!hits.length) {
|
||||
el.innerHTML += '<div class="empty">No results found.</div>';
|
||||
return;
|
||||
}
|
||||
hits.forEach(function(h) {
|
||||
const parts = (h.chunk_text || '').split('—');
|
||||
const name = (parts[0] || '').trim();
|
||||
const rest = (parts[1] || '').trim();
|
||||
const roleMatch = rest.match(/^(.+?) in (.+?)\./);
|
||||
const skillsMatch = rest.match(/Skills: ([^.]+)/);
|
||||
const certsMatch = rest.match(/Certs?: ([^.]+)/);
|
||||
const relMatch = rest.match(/Reliability: ([\d.]+)/);
|
||||
const availMatch = rest.match(/Availability: ([\d.]+)/);
|
||||
const archMatch = rest.match(/Archetype: (\w+)/);
|
||||
const w = {
|
||||
name: name, role: roleMatch ? roleMatch[1].trim() : '',
|
||||
city: roleMatch ? roleMatch[2].split(',')[0].trim() : '',
|
||||
state: roleMatch ? (roleMatch[2].split(',')[1] || '').trim() : '',
|
||||
skills: skillsMatch ? skillsMatch[1] : '', certifications: certsMatch ? certsMatch[1] : '',
|
||||
reliability: relMatch ? relMatch[1] : '', availability: availMatch ? availMatch[1] : '',
|
||||
archetype: archMatch ? archMatch[1] : '', worker_id: h.doc_id
|
||||
};
|
||||
el.appendChild(makeWorkerCard(w, h.score));
|
||||
});
|
||||
}
|
||||
|
||||
function renderHybridResults(data, query, ms) {
|
||||
const el = document.getElementById('results');
|
||||
const sources = data.sources || [];
|
||||
el.innerHTML = '';
|
||||
|
||||
const header = document.createElement('div');
|
||||
header.className = 'result-header';
|
||||
const badges = '<span class="badge green">' + (data.sql_matches || 0) + ' SQL matches</span> ' +
|
||||
'<span class="badge purple">AI ranked top ' + sources.length + '</span> ' +
|
||||
'<span class="badge blue">' + ms + 'ms</span>';
|
||||
header.innerHTML = '<span class="count">' + sources.length + ' results</span><span class="meta">' + badges + '</span>';
|
||||
el.appendChild(header);
|
||||
|
||||
if (!sources.length) {
|
||||
el.innerHTML += '<div class="empty">No matches with current filters. Try broadening the state or role filter.</div>';
|
||||
return;
|
||||
}
|
||||
sources.forEach(function(s) {
|
||||
const parts = (s.chunk_text || '').split('—');
|
||||
const name = (parts[0] || '').trim();
|
||||
const rest = (parts[1] || '').trim();
|
||||
const roleMatch = rest.match(/^(.+?) in (.+?)\./);
|
||||
const skillsMatch = rest.match(/Skills: ([^.]+)/);
|
||||
const certsMatch = rest.match(/Certs?: ([^.]+)/);
|
||||
const relMatch = rest.match(/Reliability: ([\d.]+)/);
|
||||
const availMatch = rest.match(/Availability: ([\d.]+)/);
|
||||
const w = {
|
||||
name: name, role: roleMatch ? roleMatch[1].trim() : '',
|
||||
city: roleMatch ? roleMatch[2].split(',')[0].trim() : '',
|
||||
state: roleMatch ? (roleMatch[2].split(',')[1] || '').trim() : '',
|
||||
skills: skillsMatch ? skillsMatch[1] : '', certifications: certsMatch ? certsMatch[1] : '',
|
||||
reliability: relMatch ? relMatch[1] : '', availability: availMatch ? availMatch[1] : '',
|
||||
worker_id: s.doc_id
|
||||
};
|
||||
const card = makeWorkerCard(w, s.score);
|
||||
if (s.sql_verified) {
|
||||
const v = document.createElement('div');
|
||||
v.className = 'verified';
|
||||
v.textContent = '✓ SQL verified against database';
|
||||
card.appendChild(v);
|
||||
}
|
||||
el.appendChild(card);
|
||||
});
|
||||
}
|
||||
|
||||
function makeWorkerCard(w, score) {
|
||||
const card = document.createElement('div');
|
||||
card.className = 'worker';
|
||||
|
||||
const top = document.createElement('div');
|
||||
top.className = 'top';
|
||||
|
||||
const info = document.createElement('div');
|
||||
const nameEl = document.createElement('div');
|
||||
nameEl.className = 'name';
|
||||
nameEl.textContent = w.name || w.worker_id || '—';
|
||||
info.appendChild(nameEl);
|
||||
if (w.role) { const r = document.createElement('div'); r.className = 'role'; r.textContent = w.role; info.appendChild(r); }
|
||||
if (w.city || w.state) { const l = document.createElement('div'); l.className = 'location'; l.textContent = [w.city, w.state].filter(Boolean).join(', '); info.appendChild(l); }
|
||||
top.appendChild(info);
|
||||
|
||||
if (score !== null && score !== undefined) {
|
||||
const sc = document.createElement('div');
|
||||
sc.className = 'score';
|
||||
const num = document.createElement('div');
|
||||
num.className = 'num';
|
||||
num.textContent = (score * 100).toFixed(0) + '%';
|
||||
sc.appendChild(num);
|
||||
const label = document.createElement('div');
|
||||
label.className = 'label';
|
||||
label.textContent = 'AI match';
|
||||
sc.appendChild(label);
|
||||
top.appendChild(sc);
|
||||
}
|
||||
card.appendChild(top);
|
||||
|
||||
const details = document.createElement('div');
|
||||
details.className = 'details';
|
||||
if (w.skills) addDetail(details, 'Skills', w.skills.replace(/\|/g, ', '));
|
||||
if (w.certifications && w.certifications !== 'none') addDetail(details, 'Certs', w.certifications.replace(/\|/g, ', '));
|
||||
if (w.reliability) addDetail(details, 'Reliability', w.reliability);
|
||||
if (w.availability) addDetail(details, 'Availability', w.availability);
|
||||
if (w.archetype) addDetail(details, 'Type', w.archetype);
|
||||
card.appendChild(details);
|
||||
|
||||
return card;
|
||||
}
|
||||
|
||||
function addDetail(parent, label, value) {
|
||||
const d = document.createElement('div');
|
||||
d.className = 'detail';
|
||||
const dt = document.createElement('div');
|
||||
dt.className = 'dt';
|
||||
dt.textContent = label;
|
||||
d.appendChild(dt);
|
||||
const dd = document.createElement('div');
|
||||
dd.className = 'dd';
|
||||
dd.textContent = value;
|
||||
d.appendChild(dd);
|
||||
parent.appendChild(d);
|
||||
}
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1 @@
|
||||
// Replaced by inline script in dashboard.html
|
||||
+2519
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,174 @@
|
||||
// langfuse_bridge — the missing piece called out in project_lost_stack.md
|
||||
// and Phase 40 PRD. Polls Langfuse `/api/public/traces` at interval,
|
||||
// forwards every completed trace to observer `:3800/event` with
|
||||
// `source: "langfuse"`. Observer's existing ring buffer + analyzer
|
||||
// pick it up, so the KB learns from cost/latency/provider deltas per
|
||||
// model — not just from scenario outcomes.
|
||||
//
|
||||
// Loopback: observer persistOp() appends to data/_observer/ops.jsonl
|
||||
// and its aggregator produces pathway_recommendations.jsonl. This
|
||||
// bridge closes the feedback loop between LLM call metadata and the
|
||||
// playbook/KB learning surface.
|
||||
//
|
||||
// State persistence: last-seen trace timestamp written to a JSON file
|
||||
// so restarts don't double-emit. Bounded forward window (50/tick) so
|
||||
// first-run catch-up doesn't hammer the observer.
|
||||
|
||||
const LANGFUSE_URL = process.env.LANGFUSE_URL ?? "http://localhost:3000";
|
||||
const LANGFUSE_PUBLIC = process.env.LANGFUSE_PUBLIC_KEY;
|
||||
const LANGFUSE_SECRET = process.env.LANGFUSE_SECRET_KEY;
|
||||
const OBSERVER_URL = process.env.OBSERVER_URL ?? "http://localhost:3800";
|
||||
const POLL_INTERVAL_MS = Number(process.env.LANGFUSE_POLL_MS ?? 30000);
|
||||
const BATCH_LIMIT = Number(process.env.LANGFUSE_BATCH_LIMIT ?? 50);
|
||||
const STATE_FILE = process.env.LANGFUSE_STATE_FILE
|
||||
?? "/var/lib/lakehouse-guard/langfuse_last_seen.json";
|
||||
|
||||
interface LangfuseTrace {
|
||||
id: string;
|
||||
name?: string;
|
||||
timestamp: string;
|
||||
input?: any;
|
||||
output?: any;
|
||||
latency?: number; // seconds, per Langfuse API
|
||||
totalCost?: number;
|
||||
usage?: { input?: number; output?: number; total?: number };
|
||||
metadata?: any;
|
||||
}
|
||||
|
||||
interface State { last_seen_ts?: string }
|
||||
|
||||
function basicAuth(): string {
|
||||
return "Basic " + btoa(`${LANGFUSE_PUBLIC}:${LANGFUSE_SECRET}`);
|
||||
}
|
||||
|
||||
async function loadState(): Promise<State> {
|
||||
try {
|
||||
const f = Bun.file(STATE_FILE);
|
||||
if (!(await f.exists())) return {};
|
||||
return JSON.parse(await f.text()) as State;
|
||||
} catch (e) {
|
||||
console.warn(`[langfuse-bridge] state load failed: ${e}`);
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
async function saveState(s: State): Promise<void> {
|
||||
try {
|
||||
await Bun.write(STATE_FILE, JSON.stringify(s));
|
||||
} catch (e) {
|
||||
console.warn(`[langfuse-bridge] state save failed: ${e}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function fetchTracesSince(cursor?: string): Promise<LangfuseTrace[]> {
|
||||
const url = new URL("/api/public/traces", LANGFUSE_URL);
|
||||
url.searchParams.set("limit", String(BATCH_LIMIT));
|
||||
url.searchParams.set("orderBy", "timestamp.asc");
|
||||
if (cursor) url.searchParams.set("fromTimestamp", cursor);
|
||||
const resp = await fetch(url, {
|
||||
headers: { authorization: basicAuth() },
|
||||
signal: AbortSignal.timeout(10_000),
|
||||
});
|
||||
if (!resp.ok) {
|
||||
throw new Error(`langfuse ${resp.status}: ${(await resp.text()).slice(0, 200)}`);
|
||||
}
|
||||
const body: any = await resp.json();
|
||||
return (body.data ?? []) as LangfuseTrace[];
|
||||
}
|
||||
|
||||
// Shape one Langfuse trace into the ObservedOp the observer expects
|
||||
// (see mcp-server/observer.ts:29). `source: "langfuse"` is the
|
||||
// provenance flag so the analyzer can weight traces differently from
|
||||
// scenario-sourced events.
|
||||
function toObservedOp(t: LangfuseTrace): Record<string, any> {
|
||||
const endpoint = t.metadata?.provider
|
||||
?? t.metadata?.model
|
||||
?? t.name
|
||||
?? "langfuse.trace";
|
||||
const inputSummary = typeof t.input === "string"
|
||||
? t.input.slice(0, 200)
|
||||
: JSON.stringify(t.input ?? {}).slice(0, 200);
|
||||
const outputSummary = typeof t.output === "string"
|
||||
? t.output.slice(0, 200)
|
||||
: JSON.stringify(t.output ?? {}).slice(0, 200);
|
||||
return {
|
||||
timestamp: t.timestamp,
|
||||
endpoint: `langfuse:${endpoint}`,
|
||||
input_summary: inputSummary,
|
||||
success: !t.metadata?.error,
|
||||
duration_ms: Math.round((t.latency ?? 0) * 1000),
|
||||
output_summary: outputSummary,
|
||||
source: "langfuse",
|
||||
sig_hash: t.metadata?.sig_hash,
|
||||
event_kind: t.metadata?.task_class,
|
||||
// Extra fields the observer doesn't schema but the KB aggregator
|
||||
// can still pick up via JSON passthrough.
|
||||
model: t.metadata?.model,
|
||||
provider: t.metadata?.provider,
|
||||
prompt_tokens: t.usage?.input,
|
||||
completion_tokens: t.usage?.output,
|
||||
total_tokens: t.usage?.total,
|
||||
total_cost: t.totalCost,
|
||||
};
|
||||
}
|
||||
|
||||
async function forwardToObserver(op: Record<string, any>): Promise<void> {
|
||||
const resp = await fetch(`${OBSERVER_URL}/event`, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify(op),
|
||||
signal: AbortSignal.timeout(5_000),
|
||||
});
|
||||
if (!resp.ok) {
|
||||
throw new Error(`observer ${resp.status}: ${(await resp.text()).slice(0, 200)}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function tick(): Promise<void> {
|
||||
const state = await loadState();
|
||||
let traces: LangfuseTrace[];
|
||||
try {
|
||||
traces = await fetchTracesSince(state.last_seen_ts);
|
||||
} catch (e) {
|
||||
console.warn(`[langfuse-bridge] fetch failed: ${e}`);
|
||||
return;
|
||||
}
|
||||
if (traces.length === 0) {
|
||||
console.log(`[langfuse-bridge] no new traces since ${state.last_seen_ts ?? "start"}`);
|
||||
return;
|
||||
}
|
||||
let last = state.last_seen_ts ?? "";
|
||||
let forwarded = 0;
|
||||
for (const t of traces) {
|
||||
try {
|
||||
await forwardToObserver(toObservedOp(t));
|
||||
forwarded++;
|
||||
if (t.timestamp > last) last = t.timestamp;
|
||||
} catch (e) {
|
||||
console.warn(`[langfuse-bridge] forward ${t.id} failed: ${e}`);
|
||||
// Don't advance cursor on forward failure — retry next tick.
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (last) await saveState({ last_seen_ts: last });
|
||||
console.log(
|
||||
`[langfuse-bridge] forwarded ${forwarded}/${traces.length}, last_seen=${last}`,
|
||||
);
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
if (!LANGFUSE_PUBLIC || !LANGFUSE_SECRET) {
|
||||
console.error("LANGFUSE_PUBLIC_KEY + LANGFUSE_SECRET_KEY required");
|
||||
process.exit(1);
|
||||
}
|
||||
console.log(
|
||||
`[langfuse-bridge] polling ${LANGFUSE_URL} every ${POLL_INTERVAL_MS}ms → ${OBSERVER_URL}/event`,
|
||||
);
|
||||
await tick();
|
||||
setInterval(tick, POLL_INTERVAL_MS);
|
||||
}
|
||||
|
||||
main().catch(e => {
|
||||
console.error(`[langfuse-bridge] fatal: ${e}`);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -0,0 +1,698 @@
|
||||
/**
|
||||
* Lakehouse Observer — autonomous iteration loop.
|
||||
*
|
||||
* Runs continuously alongside the agent gateway. Watches every operation,
|
||||
* logs outcomes, detects failures, and feeds learnings back so agents
|
||||
* improve over time without retraining.
|
||||
*
|
||||
* Three loops:
|
||||
* 1. OPERATION OBSERVER — wraps gateway calls, timestamps + logs every
|
||||
* success/failure to the lakehouse
|
||||
* 2. ERROR ANALYZER — periodically reads the error log, asks a local
|
||||
* model to diagnose patterns, writes recommendations
|
||||
* 3. PLAYBOOK BUILDER — after N successful ops of the same type,
|
||||
* consolidates them into a reusable playbook entry
|
||||
*
|
||||
* This is the "third-party witness" J asked for — it watches what
|
||||
* agents do and helps them not repeat mistakes.
|
||||
*/
|
||||
|
||||
const GATEWAY = process.env.GATEWAY_URL || "http://localhost:3700";
|
||||
const LAKEHOUSE = process.env.LAKEHOUSE_URL || "http://localhost:3100";
|
||||
const CYCLE_SECS = parseInt(process.env.OBSERVER_CYCLE || "30");
|
||||
// Phase 24 — observer now listens on an HTTP port for external ops
|
||||
// (scenarios bypass the MCP:3700 layer by design). Default 3800.
|
||||
const OBSERVER_PORT = parseInt(process.env.OBSERVER_PORT || "3800");
|
||||
|
||||
// ─── Observed operation log ───
|
||||
|
||||
interface ObservedOp {
|
||||
timestamp: string;
|
||||
endpoint: string;
|
||||
input_summary: string;
|
||||
success: boolean;
|
||||
duration_ms: number;
|
||||
output_summary: string;
|
||||
error?: string;
|
||||
// Phase 24 — optional provenance so error analyzer and playbook
|
||||
// builder can differentiate MCP-layer ops from scenario-sourced
|
||||
// events. Scenarios set source="scenario" + staffer_id + sig_hash.
|
||||
source?: "mcp" | "scenario" | "langfuse" | "overseer_correction";
|
||||
staffer_id?: string;
|
||||
sig_hash?: string;
|
||||
event_kind?: string;
|
||||
role?: string;
|
||||
city?: string;
|
||||
state?: string;
|
||||
count?: number;
|
||||
rescue_attempted?: boolean;
|
||||
rescue_succeeded?: boolean;
|
||||
// Overseer-correction-specific (2026-04-23): lets the analyzer
|
||||
// correlate corrections with the drift that prompted them and with
|
||||
// subsequent outcomes that either validated or invalidated the advice.
|
||||
task_class?: string;
|
||||
correction?: string;
|
||||
applied_at_turn?: number;
|
||||
}
|
||||
|
||||
const recentOps: ObservedOp[] = [];
|
||||
|
||||
// Phase 24 — external ingest path. Scenarios POST outcome summaries
|
||||
// here so the observer's analyzer + playbook builder see them. Called
|
||||
// from the Bun.serve() handler below. Same ring buffer as the MCP-
|
||||
// wrapped path so downstream loops don't need to know the source.
|
||||
export function recordExternalOp(op: ObservedOp): void {
|
||||
recentOps.push({ ...op, source: op.source ?? "scenario" });
|
||||
if (recentOps.length > 2000) recentOps.shift();
|
||||
}
|
||||
|
||||
// ─── Wrapped gateway caller — every call gets observed ───
|
||||
|
||||
export async function observed(
|
||||
endpoint: string,
|
||||
body: any,
|
||||
description: string,
|
||||
): Promise<{ data: any; op: ObservedOp }> {
|
||||
const t0 = Date.now();
|
||||
let data: any;
|
||||
let error: string | undefined;
|
||||
let success = true;
|
||||
|
||||
try {
|
||||
const resp = await fetch(`${GATEWAY}${endpoint}`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
data = await resp.json();
|
||||
if (data.error) {
|
||||
success = false;
|
||||
error = data.error;
|
||||
}
|
||||
} catch (e: any) {
|
||||
success = false;
|
||||
error = e.message;
|
||||
data = { error: e.message };
|
||||
}
|
||||
|
||||
const op: ObservedOp = {
|
||||
timestamp: new Date().toISOString(),
|
||||
endpoint,
|
||||
input_summary: description,
|
||||
success,
|
||||
duration_ms: Date.now() - t0,
|
||||
output_summary: success
|
||||
? summarize(data)
|
||||
: `ERROR: ${error}`,
|
||||
error,
|
||||
};
|
||||
|
||||
recentOps.push(op);
|
||||
if (recentOps.length > 1000) recentOps.shift();
|
||||
|
||||
// Persist to lakehouse
|
||||
await persistOp(op);
|
||||
|
||||
return { data, op };
|
||||
}
|
||||
|
||||
function summarize(data: any): string {
|
||||
if (data.sql_matches !== undefined) return `hybrid: ${data.sql_matches} sql → ${data.vector_reranked} results`;
|
||||
if (data.rows) return `${data.row_count || data.rows.length} rows`;
|
||||
if (data.answer) return `answer: ${data.answer.slice(0, 80)}...`;
|
||||
if (data.sources) return `${data.sources.length} sources`;
|
||||
return JSON.stringify(data).slice(0, 100);
|
||||
}
|
||||
|
||||
// Phase 24 honesty fix — the old persistOp used /ingest/file which
|
||||
// REPLACES the dataset (flagged in feedback_ingest_replace_semantics.md).
|
||||
// Every op silently wiped all prior ops. Now we append a JSONL line to
|
||||
// data/_observer/ops.jsonl so the historical trace is durable. The
|
||||
// observer analyzer + playbook builder read from this file when it
|
||||
// outgrows the 2000-entry in-memory ring.
|
||||
async function persistOp(op: ObservedOp) {
|
||||
try {
|
||||
const { mkdir, appendFile } = await import("node:fs/promises");
|
||||
await mkdir("data/_observer", { recursive: true });
|
||||
await appendFile("data/_observer/ops.jsonl", JSON.stringify(op) + "\n");
|
||||
} catch {
|
||||
// Persistence is best-effort; in-memory ring still works.
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
// ─── LLM Team escalation (code_review mode) ───
|
||||
//
|
||||
// When recent failures on a single sig_hash cross a threshold the
|
||||
// local qwen2.5 analysis is probably insufficient. J's 2026-04-24
|
||||
// direction: "the observer would trigger to give more context" —
|
||||
// route failure clusters to LLM Team's specialized code_review mode
|
||||
// (via /api/run) so richer structured signal lands in the KB for
|
||||
// scrum + auditor + playbook memory to consume next pass.
|
||||
//
|
||||
// Non-destructive: runs in parallel to the existing qwen2.5 analysis,
|
||||
// never replaces it. Writes to data/_kb/observer_escalations.jsonl
|
||||
// as a dedicated audit surface.
|
||||
|
||||
const LLM_TEAM = process.env.LH_LLM_TEAM_URL ?? "http://localhost:5000";
|
||||
const LLM_TEAM_ESCALATIONS = "/home/profit/lakehouse/data/_kb/observer_escalations.jsonl";
|
||||
const ESCALATION_THRESHOLD = 3; // N+ failures on same sig_hash triggers
|
||||
|
||||
async function escalateFailureClusterToLLMTeam(sigHash: string, cluster: ObservedOp[]) {
|
||||
// Package the failure cluster as a single context blob. Originally
|
||||
// I routed this to LLM Team's `code_review` mode at /api/run, but
|
||||
// that mode isn't registered in llm_team_ui.py — it returned
|
||||
// "Unknown mode" on every call. Revised 2026-04-24: route directly
|
||||
// to the gateway's /v1/chat with provider=ollama_cloud + qwen3-coder:480b
|
||||
// (the coding specialist that's rung 2 of the scrum ladder, proven
|
||||
// to produce substantive structured reviews). Fire-and-forget so
|
||||
// downstream failures don't block observer's normal loop.
|
||||
const context = cluster.slice(-8).map((o, i) =>
|
||||
`[${i + 1}] endpoint=${o.endpoint} input=${o.input_summary} error=${o.error ?? "?"}`
|
||||
).join("\n");
|
||||
const prompt = `sig_hash=${sigHash} · ${cluster.length} failures on the same signature:\n\n${context}\n\nReview this failure cluster. Identify:\n1. Likely root cause (single sentence).\n2. Files most likely responsible (path hints).\n3. Concrete fix direction (under 3 sentences).\n4. Confidence: NN%\n\nBe specific, not generic.`;
|
||||
|
||||
try {
|
||||
const resp = await fetch(`${LAKEHOUSE}/v1/chat`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
provider: "ollama_cloud",
|
||||
model: "qwen3-coder:480b",
|
||||
messages: [{ role: "user", content: prompt }],
|
||||
max_tokens: 800,
|
||||
temperature: 0.2,
|
||||
}),
|
||||
signal: AbortSignal.timeout(60000),
|
||||
});
|
||||
if (!resp.ok) {
|
||||
console.error(`[observer] escalation /v1/chat ${resp.status}: ${(await resp.text()).slice(0, 200)}`);
|
||||
return;
|
||||
}
|
||||
const j: any = await resp.json();
|
||||
const analysis = j?.choices?.[0]?.message?.content ?? "";
|
||||
|
||||
// Audit row stays schema-compatible with the prior implementation —
|
||||
// downstream consumers see structured fields regardless of the
|
||||
// review-source change. Facts/entities stay empty (this call is
|
||||
// direct-model, not extract-mode); the raw analysis carries the
|
||||
// signal.
|
||||
const row = {
|
||||
ts: new Date().toISOString(),
|
||||
source: "observer_escalation",
|
||||
mode: "direct_chat_qwen3_coder_480b",
|
||||
sig_hash: sigHash,
|
||||
cluster_size: cluster.length,
|
||||
cluster_staffer: cluster[0]?.staffer_id,
|
||||
cluster_endpoint: cluster[0]?.endpoint,
|
||||
prompt_tokens: j?.usage?.prompt_tokens ?? 0,
|
||||
completion_tokens: j?.usage?.completion_tokens ?? 0,
|
||||
analysis: analysis.slice(0, 4000),
|
||||
};
|
||||
const { appendFile } = await import("node:fs/promises");
|
||||
await appendFile(LLM_TEAM_ESCALATIONS, JSON.stringify(row) + "\n");
|
||||
console.error(
|
||||
`[observer] escalated sig_hash=${sigHash.slice(0, 8)} · cluster=${cluster.length} · ${analysis.length} chars`
|
||||
);
|
||||
} catch (e) {
|
||||
console.error(`[observer] escalation failed: ${(e as Error).message}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Track which sig_hashes we've already escalated this session so we
|
||||
// don't hammer LLM Team on every analyzeErrors tick when a cluster
|
||||
// persists across cycles.
|
||||
const escalatedSigHashes = new Set<string>();
|
||||
|
||||
// ─── Hand-review for scrum/agent candidate responses (2026-04-25) ───
|
||||
//
|
||||
// Observer is OUTSIDE the scrum loop's epistemic scope, so its verdict
|
||||
// can be treated as truth about whether a candidate review is grounded.
|
||||
// Two-tier evaluator:
|
||||
// 1. Try cloud LLM (qwen3-coder:480b) — semantic judgment with
|
||||
// response + source excerpt + grounding stats as context.
|
||||
// 2. On cloud failure (throttle/timeout) → deterministic heuristic
|
||||
// over grounding_pct + total_quotes. Marked source: "heuristic"
|
||||
// so consumers can tell which rung produced the verdict.
|
||||
// Every verdict is persisted to data/_kb/observer_reviews.jsonl.
|
||||
|
||||
const OBSERVER_REVIEWS = "/home/profit/lakehouse/data/_kb/observer_reviews.jsonl";
|
||||
|
||||
interface HandReviewInput {
|
||||
file_path: string;
|
||||
model: string;
|
||||
response: string;
|
||||
source_content: string;
|
||||
grounding_stats: { total: number; grounded: number; groundedPct: number | null };
|
||||
attempt: number;
|
||||
}
|
||||
|
||||
interface HandReviewVerdict {
|
||||
verdict: "accept" | "reject" | "cycle";
|
||||
confidence: number;
|
||||
notes: string;
|
||||
source: "cloud" | "heuristic";
|
||||
}
|
||||
|
||||
async function handReview(input: HandReviewInput): Promise<HandReviewVerdict> {
|
||||
const t0 = Date.now();
|
||||
let verdict: HandReviewVerdict;
|
||||
|
||||
try {
|
||||
verdict = await cloudHandReview(input);
|
||||
} catch (e) {
|
||||
console.error(`[observer/review] cloud failed (${(e as Error).message}); using heuristic`);
|
||||
verdict = heuristicHandReview(input);
|
||||
}
|
||||
|
||||
// Persist regardless of source so we can later compare cloud vs
|
||||
// heuristic verdicts on the same input and tune the heuristic.
|
||||
const row = {
|
||||
ts: new Date().toISOString(),
|
||||
file_path: input.file_path,
|
||||
model: input.model,
|
||||
attempt: input.attempt,
|
||||
response_chars: input.response.length,
|
||||
grounding_stats: input.grounding_stats,
|
||||
verdict: verdict.verdict,
|
||||
confidence: verdict.confidence,
|
||||
notes: verdict.notes,
|
||||
source: verdict.source,
|
||||
duration_ms: Date.now() - t0,
|
||||
};
|
||||
try {
|
||||
const { appendFile } = await import("node:fs/promises");
|
||||
await appendFile(OBSERVER_REVIEWS, JSON.stringify(row) + "\n");
|
||||
} catch { /* best-effort persistence */ }
|
||||
|
||||
return verdict;
|
||||
}
|
||||
|
||||
async function cloudHandReview(input: HandReviewInput): Promise<HandReviewVerdict> {
|
||||
const grounded = input.grounding_stats.grounded;
|
||||
const total = input.grounding_stats.total;
|
||||
const pct = input.grounding_stats.groundedPct;
|
||||
// Truncate to keep the prompt under typical context windows.
|
||||
// 2000 + 4000 = ~6000 chars ≈ 1500 tokens, plus response context.
|
||||
const responseExcerpt = input.response.slice(0, 2000);
|
||||
const sourceExcerpt = input.source_content.slice(0, 4000);
|
||||
|
||||
const prompt = `You are a code-review quality observer. Decide whether the following automated review is grounded in the actual source — not invented, not hallucinated.
|
||||
|
||||
FILE: ${input.file_path}
|
||||
MODEL: ${input.model}
|
||||
ATTEMPT: ${input.attempt}
|
||||
ANCHOR GROUNDING: ${grounded}/${total} backtick-quoted snippets matched the source verbatim${pct !== null ? ` (${pct}%)` : ""}
|
||||
|
||||
REVIEW (first 2000 chars):
|
||||
\`\`\`
|
||||
${responseExcerpt}
|
||||
\`\`\`
|
||||
|
||||
SOURCE EXCERPT (first 4000 chars):
|
||||
\`\`\`
|
||||
${sourceExcerpt}
|
||||
\`\`\`
|
||||
|
||||
Respond ONLY with a JSON object:
|
||||
{
|
||||
"verdict": "accept" | "reject" | "cycle",
|
||||
"confidence": 0-100,
|
||||
"notes": "<1-2 sentences on what makes this grounded or hallucinated>"
|
||||
}
|
||||
|
||||
- accept: review references real symbols/lines in source; findings could be acted on.
|
||||
- reject: review invents APIs, fabricates calls, contradicts source. Do NOT record.
|
||||
- cycle: review is mediocre — partially grounded but wrong shape, try a stronger model.`;
|
||||
|
||||
// Hand-review uses paid OpenRouter so it sidesteps the Ollama Cloud
|
||||
// throttle that drove every prior iter into the heuristic fallback.
|
||||
// Grok 4.1 fast: $0.20 in / $0.50 out per M tokens, 2M ctx. A typical
|
||||
// hand-review (~6K input + 300 output) costs ~$0.0014. Selected via
|
||||
// J directive 2026-04-25 ("best model under $0.72/M").
|
||||
const resp = await fetch(`${LAKEHOUSE}/v1/chat`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
provider: "openrouter",
|
||||
model: "x-ai/grok-4.1-fast",
|
||||
messages: [{ role: "user", content: prompt }],
|
||||
max_tokens: 300,
|
||||
temperature: 0.0,
|
||||
}),
|
||||
signal: AbortSignal.timeout(45000),
|
||||
});
|
||||
if (!resp.ok) {
|
||||
throw new Error(`/v1/chat ${resp.status}: ${(await resp.text()).slice(0, 200)}`);
|
||||
}
|
||||
const j: any = await resp.json();
|
||||
const content = (j?.choices?.[0]?.message?.content ?? "").trim();
|
||||
// Pull JSON object from the response — model may wrap it in prose.
|
||||
const m = content.match(/\{[\s\S]*\}/);
|
||||
if (!m) throw new Error(`no JSON object in response: ${content.slice(0, 100)}`);
|
||||
const parsed = JSON.parse(m[0]);
|
||||
const v = String(parsed.verdict ?? "accept").toLowerCase();
|
||||
return {
|
||||
verdict: (v === "reject" || v === "cycle") ? v as "reject" | "cycle" : "accept",
|
||||
confidence: Number(parsed.confidence ?? 50),
|
||||
notes: String(parsed.notes ?? "").slice(0, 500),
|
||||
source: "cloud",
|
||||
};
|
||||
}
|
||||
|
||||
function heuristicHandReview(input: HandReviewInput): HandReviewVerdict {
|
||||
// Deterministic fallback when cloud is throttled. Conservative:
|
||||
// only flip to reject when the evidence is overwhelming, otherwise
|
||||
// accept (fall-open principle — observer is policy, not blocker).
|
||||
const total = input.grounding_stats.total;
|
||||
const pct = input.grounding_stats.groundedPct;
|
||||
const respLen = input.response.length;
|
||||
|
||||
// Too short to be a real review
|
||||
if (respLen < 1500) {
|
||||
return { verdict: "reject", confidence: 80, notes: `response too short (${respLen} chars)`, source: "heuristic" };
|
||||
}
|
||||
// Below 5 quotes — not enough signal to judge grounding; accept
|
||||
if (total < 5 || pct === null) {
|
||||
return { verdict: "accept", confidence: 50, notes: `insufficient quote signal (${total} quotes); accepting`, source: "heuristic" };
|
||||
}
|
||||
// Very heavy hallucination
|
||||
if (pct < 20) {
|
||||
return { verdict: "reject", confidence: 85, notes: `low grounding (${pct}% of ${total} quotes)`, source: "heuristic" };
|
||||
}
|
||||
// Mediocre — cycle to a stronger model
|
||||
if (pct < 50) {
|
||||
return { verdict: "cycle", confidence: 65, notes: `mediocre grounding (${pct}% of ${total} quotes); try stronger`, source: "heuristic" };
|
||||
}
|
||||
// Good enough
|
||||
return { verdict: "accept", confidence: 75, notes: `grounding ${pct}% of ${total} quotes`, source: "heuristic" };
|
||||
}
|
||||
|
||||
async function maybeEscalate(failures: ObservedOp[]) {
|
||||
// Group failures by sig_hash
|
||||
const bySig = new Map<string, ObservedOp[]>();
|
||||
for (const f of failures) {
|
||||
const k = f.sig_hash ?? "__no_sig__";
|
||||
(bySig.get(k) ?? bySig.set(k, []).get(k)!).push(f);
|
||||
}
|
||||
for (const [sigHash, cluster] of bySig) {
|
||||
if (sigHash === "__no_sig__") continue;
|
||||
if (cluster.length < ESCALATION_THRESHOLD) continue;
|
||||
if (escalatedSigHashes.has(sigHash)) continue;
|
||||
escalatedSigHashes.add(sigHash);
|
||||
// Fire-and-forget — don't block the existing analyzer loop.
|
||||
escalateFailureClusterToLLMTeam(sigHash, cluster).catch(() => {});
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Error analyzer loop ───
|
||||
|
||||
async function analyzeErrors() {
|
||||
// Read recent failures
|
||||
const failures = recentOps.filter(op => !op.success);
|
||||
if (failures.length === 0) return;
|
||||
|
||||
// NEW 2026-04-24: escalate recurring sig_hash clusters to LLM Team
|
||||
// code_review mode. Runs in parallel to the local qwen2.5 analysis
|
||||
// below — non-blocking, richer downstream signal for scrum/auditor.
|
||||
maybeEscalate(failures).catch(() => {});
|
||||
|
||||
const errorSummary = failures.slice(-10).map(f =>
|
||||
`[${f.endpoint}] ${f.input_summary}: ${f.error}`
|
||||
).join("\n");
|
||||
|
||||
// Ask local model to diagnose
|
||||
try {
|
||||
const resp = await fetch(`${LAKEHOUSE}/ai/generate`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
prompt: `You are a system reliability observer. Analyze these recent failures and suggest fixes:
|
||||
|
||||
${errorSummary}
|
||||
|
||||
For each error:
|
||||
1. What likely caused it?
|
||||
2. How should the agent adjust its approach?
|
||||
3. Should this be added to the playbook as a "don't do this"?
|
||||
|
||||
Be specific and actionable. Under 200 words.`,
|
||||
model: "qwen2.5",
|
||||
max_tokens: 400,
|
||||
temperature: 0.2,
|
||||
}),
|
||||
});
|
||||
const analysis = await resp.json();
|
||||
if (analysis.text) {
|
||||
console.error(`[observer] Error analysis:\n${analysis.text}`);
|
||||
// Log the analysis as a playbook entry
|
||||
await fetch(`${GATEWAY}/log`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
operation: `error_analysis: ${failures.length} failures`,
|
||||
approach: "LLM-analyzed error patterns",
|
||||
result: analysis.text.slice(0, 500),
|
||||
context: errorSummary.slice(0, 500),
|
||||
}),
|
||||
});
|
||||
}
|
||||
} catch (e) {
|
||||
console.error(`[observer] Analysis failed: ${e}`);
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Playbook consolidation ───
|
||||
|
||||
async function consolidatePlaybooks() {
|
||||
const successes = recentOps.filter(op => op.success);
|
||||
if (successes.length < 5) return;
|
||||
|
||||
// Group by endpoint
|
||||
const groups: Record<string, ObservedOp[]> = {};
|
||||
for (const op of successes) {
|
||||
const key = op.endpoint;
|
||||
if (!groups[key]) groups[key] = [];
|
||||
groups[key].push(op);
|
||||
}
|
||||
|
||||
for (const [endpoint, ops] of Object.entries(groups)) {
|
||||
if (ops.length < 3) continue;
|
||||
|
||||
const avgDuration = ops.reduce((s, o) => s + o.duration_ms, 0) / ops.length;
|
||||
const pattern = ops.slice(-3).map(o => o.input_summary).join("; ");
|
||||
|
||||
await fetch(`${GATEWAY}/log`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
operation: `consolidated: ${ops.length} successful ${endpoint} calls`,
|
||||
approach: `common pattern: ${pattern.slice(0, 200)}`,
|
||||
result: `avg_duration=${avgDuration.toFixed(0)}ms, ${ops.length} successes`,
|
||||
context: `endpoint=${endpoint}`,
|
||||
}),
|
||||
}).catch(() => {});
|
||||
}
|
||||
}
|
||||
|
||||
// ─── HTTP listener for external ops (Phase 24) ───
|
||||
|
||||
// Scenarios POST per-event outcomes here so the observer's analyzer +
|
||||
// playbook builder see them alongside MCP-wrapped ops. Read-only stats
|
||||
// also exposed at /stats for external health checks.
|
||||
function startHttpListener() {
|
||||
Bun.serve({
|
||||
port: OBSERVER_PORT,
|
||||
hostname: "0.0.0.0",
|
||||
fetch(req) {
|
||||
const url = new URL(req.url);
|
||||
if (req.method === "GET" && url.pathname === "/health") {
|
||||
return new Response(JSON.stringify({ status: "ok", ops_in_ring: recentOps.length }));
|
||||
}
|
||||
if (req.method === "GET" && url.pathname === "/stats") {
|
||||
const bySource = new Map<string, number>();
|
||||
for (const o of recentOps) {
|
||||
const k = o.source ?? "mcp";
|
||||
bySource.set(k, (bySource.get(k) ?? 0) + 1);
|
||||
}
|
||||
return new Response(JSON.stringify({
|
||||
total: recentOps.length,
|
||||
successes: recentOps.filter(o => o.success).length,
|
||||
failures: recentOps.filter(o => !o.success).length,
|
||||
by_source: Object.fromEntries(bySource),
|
||||
recent_scenario_ops: recentOps
|
||||
.filter(o => o.source === "scenario")
|
||||
.slice(-10)
|
||||
.map(o => ({ ts: o.timestamp, ok: o.success, staffer: o.staffer_id, kind: o.event_kind, role: o.role })),
|
||||
}));
|
||||
}
|
||||
// ─── Hand-review endpoint (2026-04-25) ───
|
||||
// scrum/agent posts a candidate response + source content + grounding
|
||||
// stats. Observer evaluates via cloud LLM (qwen3-coder:480b) with
|
||||
// semantic context and returns {verdict, confidence, notes}. On
|
||||
// cloud throttle, falls back to a deterministic heuristic over the
|
||||
// grounding stats so the loop keeps moving with honest signal.
|
||||
//
|
||||
// This is the policy layer scrum was missing — pre-2026-04-25 the
|
||||
// scrum_master applied a hardcoded grounding-rate threshold inline,
|
||||
// which baked judgment into the wrong layer. Now scrum reports data
|
||||
// (response + source + stats) and observer decides accept/reject/cycle.
|
||||
if (req.method === "POST" && url.pathname === "/review") {
|
||||
return req.json().then((body: any) => handReview(body))
|
||||
.then((verdict) => new Response(JSON.stringify(verdict), {
|
||||
headers: { "content-type": "application/json" },
|
||||
}))
|
||||
.catch((e: Error) =>
|
||||
new Response(JSON.stringify({ verdict: "accept", notes: `observer error: ${e.message}`, source: "heuristic" }), {
|
||||
status: 200, // fall-open shape — scrum keeps moving on observer failure
|
||||
headers: { "content-type": "application/json" },
|
||||
}));
|
||||
}
|
||||
if (req.method === "POST" && url.pathname === "/event") {
|
||||
return req.json().then((body: any) => {
|
||||
const op: ObservedOp = {
|
||||
timestamp: body.timestamp ?? new Date().toISOString(),
|
||||
endpoint: body.endpoint ?? "scenario:fill",
|
||||
input_summary: body.input_summary ?? `${body.event_kind ?? "?"} ${body.role ?? "?"}×${body.count ?? "?"} in ${body.city ?? "?"}, ${body.state ?? "?"}`,
|
||||
success: !!body.success,
|
||||
duration_ms: Number(body.duration_ms ?? 0),
|
||||
output_summary: body.output_summary ?? (body.success ? "filled" : (body.error ?? "failed")),
|
||||
error: body.error,
|
||||
// Respect the client's provenance if set (langfuse bridge
|
||||
// sends source:"langfuse", etc.). Default to "scenario"
|
||||
// to keep legacy Phase 24 callers working.
|
||||
source: body.source ?? "scenario",
|
||||
staffer_id: body.staffer_id,
|
||||
sig_hash: body.sig_hash,
|
||||
event_kind: body.event_kind,
|
||||
role: body.role,
|
||||
city: body.city,
|
||||
state: body.state,
|
||||
count: body.count,
|
||||
rescue_attempted: !!body.rescue_attempted,
|
||||
rescue_succeeded: !!body.rescue_succeeded,
|
||||
};
|
||||
recordExternalOp(op);
|
||||
persistOp(op).catch(() => {});
|
||||
return new Response(JSON.stringify({ accepted: true, ring_size: recentOps.length }));
|
||||
}).catch((e: Error) =>
|
||||
new Response(JSON.stringify({ error: e.message }), { status: 400 }));
|
||||
}
|
||||
return new Response("not found", { status: 404 });
|
||||
},
|
||||
});
|
||||
console.error(`[observer] HTTP listener bound to 0.0.0.0:${OBSERVER_PORT}`);
|
||||
}
|
||||
|
||||
// ─── Overseer corrections tailer (2026-04-23) ───
|
||||
|
||||
// The gateway's /v1/respond loop writes T3 overseer corrections to
|
||||
// data/_kb/overseer_corrections.jsonl. Tail it once per cycle and
|
||||
// inject each new row into the same recentOps ring that analyzeErrors
|
||||
// + consolidatePlaybooks read — so a correction that just fired shows
|
||||
// up alongside the outcomes it was meant to repair, and the analyzer
|
||||
// can flag patterns like "three corrections on staffing.fill with the
|
||||
// same advice — underlying problem isn't a drift, it's a data gap".
|
||||
const CORRECTIONS_PATH = process.env.OVERSEER_CORRECTIONS_PATH
|
||||
?? "/home/profit/lakehouse/data/_kb/overseer_corrections.jsonl";
|
||||
let correctionsCursor = 0; // byte offset
|
||||
|
||||
async function tailOverseerCorrections(): Promise<number> {
|
||||
const f = Bun.file(CORRECTIONS_PATH);
|
||||
if (!(await f.exists())) return 0;
|
||||
const size = f.size;
|
||||
if (size <= correctionsCursor) return 0;
|
||||
|
||||
// Read only the suffix since the last cursor; keeps tail work
|
||||
// bounded even as the file grows.
|
||||
const text = await f.slice(correctionsCursor, size).text();
|
||||
correctionsCursor = size;
|
||||
|
||||
let forwarded = 0;
|
||||
for (const line of text.split("\n")) {
|
||||
if (!line.trim()) continue;
|
||||
let row: any;
|
||||
try { row = JSON.parse(line); } catch { continue; }
|
||||
const op: ObservedOp = {
|
||||
timestamp: row.created_at ?? new Date().toISOString(),
|
||||
endpoint: `overseer:${row.model ?? "gpt-oss:120b"}`,
|
||||
input_summary: `${row.task_class ?? "?"}: ${row.reason ?? "escalation"}`,
|
||||
// Correction itself is neither success nor failure — it's a
|
||||
// mitigation attempt. We mark success=true so analyzeErrors
|
||||
// doesn't count it as a failure, but the preview lets the
|
||||
// analyzer see what was tried.
|
||||
success: true,
|
||||
duration_ms: Number(row.usage?.latency_ms ?? 0),
|
||||
output_summary: String(row.correction ?? "").slice(0, 200),
|
||||
source: "overseer_correction",
|
||||
sig_hash: row.sig_hash,
|
||||
task_class: row.task_class,
|
||||
correction: String(row.correction ?? ""),
|
||||
applied_at_turn: Number(row.applied_at_turn ?? 0),
|
||||
};
|
||||
recordExternalOp(op);
|
||||
forwarded++;
|
||||
}
|
||||
return forwarded;
|
||||
}
|
||||
|
||||
// ─── Main loop ───
|
||||
|
||||
async function main() {
|
||||
console.error(`[observer] started — cycle=${CYCLE_SECS}s, gateway=${GATEWAY}, port=${OBSERVER_PORT}`);
|
||||
|
||||
// Run a health check first
|
||||
const health = await fetch(`${GATEWAY}/health`).then(r => r.json()).catch(() => null);
|
||||
if (!health) {
|
||||
console.error("[observer] gateway unreachable — exiting");
|
||||
process.exit(1);
|
||||
}
|
||||
console.error(`[observer] gateway healthy: ${JSON.stringify(health)}`);
|
||||
|
||||
// Phase 24 — bind HTTP listener so scenarios can POST outcomes.
|
||||
startHttpListener();
|
||||
|
||||
// Main loop
|
||||
let cycle = 0;
|
||||
while (true) {
|
||||
await Bun.sleep(CYCLE_SECS * 1000);
|
||||
cycle++;
|
||||
|
||||
// Every cycle: tail the overseer corrections KB stream, then
|
||||
// analyze errors. Order matters — if an overseer correction just
|
||||
// landed for a sig_hash that previously failed, the analyzer
|
||||
// should see both.
|
||||
const newCorrections = await tailOverseerCorrections();
|
||||
if (newCorrections > 0) {
|
||||
console.error(`[observer] pulled ${newCorrections} new overseer correction(s) into ring`);
|
||||
}
|
||||
await analyzeErrors();
|
||||
|
||||
// Every 5 cycles: consolidate playbooks
|
||||
if (cycle % 5 === 0) {
|
||||
await consolidatePlaybooks();
|
||||
}
|
||||
|
||||
const scenarioOps = recentOps.filter(o => o.source === "scenario").length;
|
||||
const langfuseOps = recentOps.filter(o => o.source === "langfuse").length;
|
||||
const correctionOps = recentOps.filter(o => o.source === "overseer_correction").length;
|
||||
const stats = {
|
||||
cycle,
|
||||
total_ops: recentOps.length,
|
||||
successes: recentOps.filter(o => o.success).length,
|
||||
failures: recentOps.filter(o => !o.success).length,
|
||||
scenario_ops: scenarioOps,
|
||||
langfuse_ops: langfuseOps,
|
||||
overseer_corrections: correctionOps,
|
||||
};
|
||||
console.error(`[observer] cycle ${cycle}: ${JSON.stringify(stats)}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Export the observed wrapper for other agents to use
|
||||
export { main as startObserver };
|
||||
|
||||
// Run if executed directly
|
||||
if (import.meta.main) {
|
||||
main().catch(console.error);
|
||||
}
|
||||
@@ -0,0 +1,414 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en"><head>
|
||||
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>Lakehouse — Connect Your Data</title>
|
||||
<style>
|
||||
*{margin:0;padding:0;box-sizing:border-box}
|
||||
body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;color:#b0b8c4;font-size:14px;line-height:1.55;-webkit-font-smoothing:antialiased}
|
||||
a{color:#58a6ff;text-decoration:none}
|
||||
a:hover{color:#79c0ff}
|
||||
|
||||
.bar{background:#0d1117;padding:0 24px;height:56px;border-bottom:1px solid #171d27;display:flex;justify-content:space-between;align-items:center;position:sticky;top:0;z-index:10}
|
||||
.bar h1{font-size:14px;font-weight:600;color:#e6edf3;letter-spacing:-0.2px}
|
||||
.bar nav{display:flex;gap:2px}
|
||||
.bar nav a{font-size:12px;color:#545d68;padding:6px 14px;border-radius:6px;transition:all 0.15s}
|
||||
.bar nav a:hover{color:#e6edf3;background:#161b22}
|
||||
.bar nav a.active{color:#e6edf3;background:#1c2333}
|
||||
.bar .rt{font-size:11px;color:#545d68}
|
||||
|
||||
.wrap{max-width:1040px;margin:0 auto;padding:28px 20px 60px}
|
||||
|
||||
.step{margin-bottom:36px;padding-left:44px;position:relative}
|
||||
.step:before{content:attr(data-n);position:absolute;left:0;top:-2px;width:30px;height:30px;border-radius:15px;background:#0d1117;border:1px solid #21262d;color:#58a6ff;font-weight:700;font-size:13px;display:flex;align-items:center;justify-content:center}
|
||||
.step.active:before{background:#1f6feb;color:#fff;border-color:#1f6feb}
|
||||
.step.done:before{content:"✓";background:#0d2818;color:#3fb950;border-color:#2ea043}
|
||||
.step h2{color:#e6edf3;font-size:18px;font-weight:700;margin-bottom:6px;letter-spacing:-0.2px}
|
||||
.step .lede{color:#8b949e;font-size:13px;margin-bottom:14px;line-height:1.6}
|
||||
|
||||
.card{background:#0d1117;border:1px solid #171d27;border-radius:10px;padding:18px;margin:8px 0}
|
||||
|
||||
.btn{padding:10px 20px;background:#1f6feb;border:none;border-radius:8px;color:#fff;font-size:13px;font-weight:600;cursor:pointer;text-decoration:none;display:inline-block}
|
||||
.btn:hover{background:#388bfd}
|
||||
.btn:disabled{opacity:0.4;cursor:not-allowed}
|
||||
.btn.ghost{background:transparent;border:1px solid #21262d;color:#c9d1d9}
|
||||
.btn.ghost:hover{background:#161b22;border-color:#30363d}
|
||||
.btn.green{background:#2ea043}
|
||||
.btn.green:hover{background:#3fb950}
|
||||
|
||||
.drop{border:2px dashed #30363d;border-radius:10px;padding:28px;text-align:center;transition:all 0.15s;cursor:pointer}
|
||||
.drop:hover{border-color:#58a6ff;background:#0d1117}
|
||||
.drop.active{border-color:#2ea043;background:#0d2818}
|
||||
.drop input[type=file]{display:none}
|
||||
.drop .title{color:#e6edf3;font-weight:600;margin-bottom:4px}
|
||||
.drop .sub{color:#545d68;font-size:12px}
|
||||
|
||||
.actions{display:flex;gap:10px;flex-wrap:wrap;margin-top:10px}
|
||||
|
||||
input[type=text]{padding:10px 14px;background:#161b22;border:1px solid #21262d;border-radius:8px;color:#e6edf3;font-size:13px;outline:none;width:100%}
|
||||
input[type=text]:focus{border-color:#388bfd}
|
||||
|
||||
table.preview{width:100%;border-collapse:collapse;font-size:11px;margin-top:8px;font-family:ui-monospace,Menlo,monospace}
|
||||
table.preview th{text-align:left;padding:8px 10px;color:#8b949e;background:#0d1117;border-bottom:1px solid #21262d;font-weight:600;font-size:10px;text-transform:uppercase;letter-spacing:0.8px}
|
||||
table.preview td{padding:6px 10px;border-bottom:1px solid #171d27;color:#c9d1d9;white-space:nowrap;max-width:180px;overflow:hidden;text-overflow:ellipsis}
|
||||
table.preview tr:hover td{background:#0d1117}
|
||||
|
||||
.col-list{display:flex;flex-direction:column;gap:6px;margin-top:6px}
|
||||
.col{display:flex;align-items:center;gap:8px;padding:8px 12px;background:#161b22;border-radius:6px;font-size:12px}
|
||||
.col .name{color:#e6edf3;font-weight:500;min-width:140px}
|
||||
.col .type{color:#58a6ff;font-size:11px;font-family:ui-monospace,Menlo,monospace;min-width:60px}
|
||||
.col .pii{padding:2px 8px;border-radius:9px;font-size:9px;font-weight:600;letter-spacing:0.3px}
|
||||
.pii.email{background:#2d1b00;color:#d29922;border:1px solid #854d0e}
|
||||
.pii.phone{background:#2d1b00;color:#d29922;border:1px solid #854d0e}
|
||||
.pii.ssn{background:#3a1a1a;color:#f85149;border:1px solid #7f1d1d}
|
||||
.pii.addr{background:#2d1b00;color:#d29922;border:1px solid #854d0e}
|
||||
.col .sample{color:#8b949e;font-size:11px;font-family:ui-monospace,Menlo,monospace;flex:1;min-width:0;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}
|
||||
|
||||
.note{color:#8b949e;font-size:12px;line-height:1.7;padding:10px 14px;border-left:2px solid #21262d;margin:10px 0}
|
||||
.note strong{color:#c9d1d9}
|
||||
|
||||
.result{padding:16px;border-radius:10px;margin:10px 0}
|
||||
.result.ok{background:#0d2818;border:1px solid #2ea04360;color:#86efac}
|
||||
.result.err{background:#3a1a1a;border:1px solid #f8514960;color:#fca5a5}
|
||||
.result .big{font-size:16px;font-weight:700;margin-bottom:4px}
|
||||
.result .detail{font-size:12px}
|
||||
|
||||
.spin{display:inline-block;width:14px;height:14px;border:2px solid #30363d;border-top-color:#58a6ff;border-radius:50%;animation:spin 0.7s linear infinite;vertical-align:middle;margin-right:6px}
|
||||
@keyframes spin{to{transform:rotate(360deg)}}
|
||||
|
||||
.footer{border-top:1px solid #171d27;padding:20px;text-align:center;color:#3d444d;font-size:11px}
|
||||
|
||||
@media(max-width:720px){
|
||||
.wrap{padding:20px 12px 40px}
|
||||
.step{padding-left:36px}
|
||||
.step:before{width:26px;height:26px;font-size:12px}
|
||||
.bar nav{display:none}
|
||||
}
|
||||
</style></head>
|
||||
<body>
|
||||
|
||||
<div class="bar">
|
||||
<h1>Lakehouse — Connect Your Data</h1>
|
||||
<nav>
|
||||
<a href=".">Dashboard</a>
|
||||
<a href="console">Walkthrough</a>
|
||||
<a href="proof">Architecture</a>
|
||||
<a href="spec">Spec</a>
|
||||
<a href="onboard" class="active">Onboard</a>
|
||||
<a href="alerts">Alerts</a>
|
||||
<a href="workspaces">Workspaces</a>
|
||||
</nav>
|
||||
<div class="rt">30 minutes from CSV to live search</div>
|
||||
</div>
|
||||
|
||||
<div class="wrap">
|
||||
|
||||
<div style="margin-bottom:24px">
|
||||
<h1 style="color:#e6edf3;font-size:26px;font-weight:700;letter-spacing:-0.5px;margin-bottom:8px">Ingest any staffing CSV in three steps</h1>
|
||||
<p style="color:#8b949e;font-size:14px;line-height:1.6;max-width:720px">
|
||||
Upload your ATS export, your worker roster, or any CSV with a <code>name</code> column.
|
||||
The wizard auto-detects columns, flags PII, previews the first rows, then ingests
|
||||
as a queryable Parquet dataset. Everything that follows — hybrid search,
|
||||
playbook ranking, pattern discovery — works against your data automatically.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<!-- Step 1 -->
|
||||
<div class="step active" data-n="1" id="step-1">
|
||||
<h2>Pick a file</h2>
|
||||
<div class="lede">Drag a CSV in, pick from disk, or use the sample roster to see the flow without any real data.</div>
|
||||
<label class="drop" id="drop">
|
||||
<input type="file" id="file" accept=".csv,text/csv">
|
||||
<div class="title">Drop a CSV here or click to choose</div>
|
||||
<div class="sub">Max 50MB · UTF-8 · comma-separated with a header row</div>
|
||||
</label>
|
||||
<div class="actions">
|
||||
<a class="btn ghost" href="samples/staffing_roster_sample.csv" download>Download sample roster (25 workers)</a>
|
||||
<button class="btn ghost" id="use-sample" type="button">Use the sample as my data</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Step 2 -->
|
||||
<div class="step" data-n="2" id="step-2" style="display:none">
|
||||
<h2>Review what we detected</h2>
|
||||
<div class="lede">
|
||||
Columns auto-typed. PII columns flagged. First rows previewed. Nothing is written to the system yet — this is a read-only dry-run.
|
||||
</div>
|
||||
<div class="card">
|
||||
<div style="color:#545d68;font-size:11px;text-transform:uppercase;letter-spacing:1.2px;font-weight:600;margin-bottom:10px">Columns detected</div>
|
||||
<div class="col-list" id="col-list"></div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<div style="color:#545d68;font-size:11px;text-transform:uppercase;letter-spacing:1.2px;font-weight:600;margin-bottom:10px">First rows</div>
|
||||
<div style="overflow-x:auto"><table class="preview" id="preview-table"></table></div>
|
||||
<div style="color:#545d68;font-size:11px;margin-top:8px" id="row-count"></div>
|
||||
</div>
|
||||
<div class="note">
|
||||
<strong>What happens next.</strong> On commit, the file is sent to <code>/ingest/file</code> — the same
|
||||
endpoint every other ingest path uses. The Rust gateway writes it to object storage as
|
||||
Parquet, computes a schema fingerprint, registers it in the catalog, and auto-detects PII
|
||||
columns server-side. Re-uploading the same file is a no-op (deduplicated by content hash).
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Step 3 -->
|
||||
<div class="step" data-n="3" id="step-3" style="display:none">
|
||||
<h2>Name it and commit</h2>
|
||||
<div class="lede">Give the dataset a queryable name. This becomes the table you can <code>SELECT * FROM</code> immediately after commit.</div>
|
||||
<div class="card">
|
||||
<label style="display:block;color:#8b949e;font-size:12px;margin-bottom:6px">Dataset name</label>
|
||||
<input type="text" id="dataset-name" placeholder="e.g. acme_staffing_roster" value="client_workers">
|
||||
<div style="color:#545d68;font-size:11px;margin-top:6px">
|
||||
Use lowercase + underscores. Once committed: queryable via <code>/query/sql</code>,
|
||||
searchable via <code>/search</code>, indexable via <code>/vectors/index</code>.
|
||||
</div>
|
||||
<div class="actions" style="margin-top:14px">
|
||||
<button class="btn green" id="commit-btn" type="button">Commit dataset</button>
|
||||
<button class="btn ghost" id="back-btn" type="button">Back</button>
|
||||
</div>
|
||||
</div>
|
||||
<div id="commit-result"></div>
|
||||
</div>
|
||||
|
||||
<!-- Step 4: Post-commit next steps (hidden until success) -->
|
||||
<div class="step" data-n="4" id="step-4" style="display:none">
|
||||
<h2>Your dataset is live</h2>
|
||||
<div class="lede">From here, the rest of the system applies to your data with zero additional setup.</div>
|
||||
<div class="card">
|
||||
<div style="color:#e6edf3;font-size:14px;font-weight:600;margin-bottom:12px">What you can do right now</div>
|
||||
<div style="display:flex;flex-direction:column;gap:10px">
|
||||
<div style="padding:10px 14px;background:#161b22;border-radius:6px;font-size:12px"><strong style="color:#e6edf3">Query via SQL.</strong> <code>POST /query/sql</code> with <code>SELECT * FROM <span id="ds-name-1">your_dataset</span> LIMIT 10</code>.</div>
|
||||
<div style="padding:10px 14px;background:#161b22;border-radius:6px;font-size:12px"><strong style="color:#e6edf3">Build a vector index.</strong> <code>POST /vectors/index</code> with <code>{"dataset":"<span id="ds-name-2">your_dataset</span>","text_column":"skills"}</code>. Embeddings stream in; queryable progressively.</div>
|
||||
<div style="padding:10px 14px;background:#161b22;border-radius:6px;font-size:12px"><strong style="color:#e6edf3">Search via the dashboard.</strong> Open the <a href=".">dashboard</a> and the "Search all workers" box. Results will come from your data.</div>
|
||||
<div style="padding:10px 14px;background:#161b22;border-radius:6px;font-size:12px"><strong style="color:#e6edf3">Track with playbook memory.</strong> Every Call/SMS/No-show click on a worker card trains the system on your data.</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<div class="footer">Lakehouse · onboarding wizard · <a href="spec">full architecture spec</a></div>
|
||||
|
||||
<script>
|
||||
var P=location.pathname.indexOf('/lakehouse')>=0?'/lakehouse':'';
|
||||
var A=location.origin+P;
|
||||
|
||||
var parsedFile=null; // { name, rows, columns, sample }
|
||||
var rawBlob=null;
|
||||
|
||||
function el(tag,cls,text){
|
||||
var e=document.createElement(tag);
|
||||
if(cls) e.className=cls;
|
||||
if(text!==undefined && text!==null) e.textContent=String(text);
|
||||
return e;
|
||||
}
|
||||
|
||||
function setStep(n){
|
||||
for(var i=1;i<=4;i++){
|
||||
var s=document.getElementById('step-'+i);
|
||||
if(!s) continue;
|
||||
s.classList.remove('active','done');
|
||||
if(i<n) s.classList.add('done');
|
||||
if(i===n) { s.classList.add('active'); s.style.display='block'; }
|
||||
else if(i>n) s.style.display='none';
|
||||
else s.style.display='block';
|
||||
}
|
||||
}
|
||||
|
||||
// Basic CSV parser. Handles quoted fields with commas + escaped quotes.
|
||||
function parseCSV(text){
|
||||
var lines=[], i=0, cur='', row=[], inQ=false;
|
||||
while(i<text.length){
|
||||
var c=text[i];
|
||||
if(inQ){
|
||||
if(c==='"' && text[i+1]==='"'){ cur+='"'; i+=2; continue; }
|
||||
if(c==='"'){ inQ=false; i++; continue; }
|
||||
cur+=c; i++;
|
||||
} else {
|
||||
if(c==='"'){ inQ=true; i++; continue; }
|
||||
if(c===','){ row.push(cur); cur=''; i++; continue; }
|
||||
if(c==='\n'){ row.push(cur); lines.push(row); row=[]; cur=''; i++; continue; }
|
||||
if(c==='\r'){ i++; continue; }
|
||||
cur+=c; i++;
|
||||
}
|
||||
}
|
||||
if(cur.length>0||row.length>0){ row.push(cur); lines.push(row); }
|
||||
return lines;
|
||||
}
|
||||
|
||||
// Type + PII inference from a sample of values for a column.
|
||||
function inferColumn(name, values){
|
||||
var nonEmpty=values.filter(function(v){return v!=null && v!==''});
|
||||
var n=nonEmpty.length;
|
||||
var lname=(name||'').toLowerCase();
|
||||
|
||||
// PII signals by name and sample content
|
||||
var pii=null;
|
||||
if(/email|e-mail|mail_address/.test(lname)) pii='email';
|
||||
else if(/phone|mobile|cell|tel/.test(lname)) pii='phone';
|
||||
else if(/ssn|social|tax_id/.test(lname)) pii='ssn';
|
||||
else if(/address|street|zip|postal/.test(lname)) pii='addr';
|
||||
// Content-based PII scan on sample
|
||||
if(!pii){
|
||||
var emailRe=/^[^\s@]+@[^\s@]+\.[^\s@]+$/;
|
||||
var phoneRe=/^[\s\-\(\)\+]*\d[\s\-\(\)\+\d]{6,}$/;
|
||||
var emailHits=nonEmpty.filter(function(v){return emailRe.test(v)}).length;
|
||||
var phoneHits=nonEmpty.filter(function(v){return phoneRe.test(v)}).length;
|
||||
if(emailHits>n*0.5) pii='email';
|
||||
else if(phoneHits>n*0.5) pii='phone';
|
||||
}
|
||||
|
||||
// Type inference
|
||||
var type='text';
|
||||
if(n>0){
|
||||
var intHits=nonEmpty.filter(function(v){return /^-?\d+$/.test(v)}).length;
|
||||
var numHits=nonEmpty.filter(function(v){return /^-?\d+(\.\d+)?$/.test(v)}).length;
|
||||
if(intHits===n) type='int';
|
||||
else if(numHits===n) type='decimal';
|
||||
else if(nonEmpty.every(function(v){return /^\d{4}-\d{2}-\d{2}/.test(v)})) type='date';
|
||||
}
|
||||
|
||||
return { name: name, type: type, pii: pii, sample: nonEmpty.slice(0,3).join(' · ') || '(empty)' };
|
||||
}
|
||||
|
||||
function readFile(file){
|
||||
rawBlob=file;
|
||||
var reader=new FileReader();
|
||||
reader.onload=function(e){
|
||||
var text=String(e.target.result||'');
|
||||
var lines=parseCSV(text);
|
||||
if(lines.length<2){
|
||||
alert('CSV is empty or has no data rows.');
|
||||
return;
|
||||
}
|
||||
var header=lines[0].map(function(h){return String(h||'').trim()});
|
||||
var rows=lines.slice(1).filter(function(r){return r.length>0 && r.some(function(c){return c!=null&&c!==''})});
|
||||
|
||||
// Build columns with inference
|
||||
var columns=header.map(function(h,idx){
|
||||
var colValues=rows.map(function(r){return r[idx]});
|
||||
return inferColumn(h, colValues.slice(0,50)); // sample first 50 for perf
|
||||
});
|
||||
|
||||
parsedFile={name: file.name, rowCount: rows.length, header: header, rows: rows, columns: columns};
|
||||
renderPreview();
|
||||
setStep(2);
|
||||
};
|
||||
reader.readAsText(file);
|
||||
}
|
||||
|
||||
function renderPreview(){
|
||||
var colList=document.getElementById('col-list'); colList.textContent='';
|
||||
parsedFile.columns.forEach(function(c){
|
||||
var row=el('div','col');
|
||||
row.appendChild(el('span','name',c.name));
|
||||
row.appendChild(el('span','type',c.type));
|
||||
if(c.pii){
|
||||
var chip=el('span','pii '+c.pii,c.pii.toUpperCase());
|
||||
row.appendChild(chip);
|
||||
}
|
||||
row.appendChild(el('span','sample',c.sample));
|
||||
colList.appendChild(row);
|
||||
});
|
||||
|
||||
var table=document.getElementById('preview-table');table.textContent='';
|
||||
var thead=document.createElement('thead');
|
||||
var htr=document.createElement('tr');
|
||||
parsedFile.header.forEach(function(h){ htr.appendChild(el('th',null,h)) });
|
||||
thead.appendChild(htr);table.appendChild(thead);
|
||||
var tbody=document.createElement('tbody');
|
||||
parsedFile.rows.slice(0,5).forEach(function(r){
|
||||
var tr=document.createElement('tr');
|
||||
parsedFile.header.forEach(function(_,idx){
|
||||
tr.appendChild(el('td',null,r[idx]||''));
|
||||
});
|
||||
tbody.appendChild(tr);
|
||||
});
|
||||
table.appendChild(tbody);
|
||||
|
||||
document.getElementById('row-count').textContent=parsedFile.rowCount.toLocaleString()+' data rows · '+parsedFile.header.length+' columns · source: '+parsedFile.name;
|
||||
|
||||
// Auto-advance button: a Next button at the bottom of step-2
|
||||
var step2=document.getElementById('step-2');
|
||||
var existing=document.getElementById('step-2-next');
|
||||
if(!existing){
|
||||
var wrap=el('div','actions');wrap.id='step-2-next';
|
||||
var next=el('button','btn','Continue to commit');next.type='button';
|
||||
next.onclick=function(){ setStep(3); };
|
||||
var reset=el('button','btn ghost','Pick a different file');reset.type='button';
|
||||
reset.onclick=function(){ parsedFile=null; rawBlob=null; document.getElementById('file').value=''; setStep(1); };
|
||||
wrap.appendChild(next);wrap.appendChild(reset);
|
||||
step2.appendChild(wrap);
|
||||
}
|
||||
}
|
||||
|
||||
function commit(){
|
||||
var dsName=(document.getElementById('dataset-name').value||'').trim();
|
||||
if(!dsName){ alert('Dataset name required.'); return; }
|
||||
if(!/^[a-z][a-z0-9_]*$/.test(dsName)){ alert('Dataset name: lowercase letters, numbers, underscores only; start with a letter.'); return; }
|
||||
if(!rawBlob){ alert('No file loaded.'); return; }
|
||||
|
||||
var btn=document.getElementById('commit-btn');
|
||||
btn.disabled=true;btn.textContent='';
|
||||
var spin=el('span','spin');btn.appendChild(spin);
|
||||
btn.appendChild(document.createTextNode('Ingesting…'));
|
||||
|
||||
var form=new FormData();
|
||||
form.append('file', rawBlob, parsedFile.name);
|
||||
|
||||
fetch(A+'/onboard/ingest?name='+encodeURIComponent(dsName), {
|
||||
method: 'POST', body: form,
|
||||
}).then(function(r){
|
||||
return r.text().then(function(body){ return {status:r.status, body:body}; });
|
||||
}).then(function(res){
|
||||
btn.disabled=false;btn.textContent='Commit dataset';
|
||||
var out=document.getElementById('commit-result');out.textContent='';
|
||||
if(res.status>=200 && res.status<300){
|
||||
var ok=el('div','result ok');
|
||||
ok.appendChild(el('div','big','Dataset "'+dsName+'" is live.'));
|
||||
ok.appendChild(el('div','detail',parsedFile.rowCount.toLocaleString()+' rows · '+parsedFile.header.length+' columns · registered in catalog'));
|
||||
out.appendChild(ok);
|
||||
document.getElementById('ds-name-1').textContent=dsName;
|
||||
document.getElementById('ds-name-2').textContent=dsName;
|
||||
setStep(4);
|
||||
} else {
|
||||
var err=el('div','result err');
|
||||
err.appendChild(el('div','big','Ingest failed ('+res.status+')'));
|
||||
err.appendChild(el('div','detail',res.body.slice(0,400)));
|
||||
out.appendChild(err);
|
||||
}
|
||||
}).catch(function(e){
|
||||
btn.disabled=false;btn.textContent='Commit dataset';
|
||||
var out=document.getElementById('commit-result');out.textContent='';
|
||||
var err=el('div','result err');
|
||||
err.appendChild(el('div','big','Network error'));
|
||||
err.appendChild(el('div','detail',e.message||String(e)));
|
||||
out.appendChild(err);
|
||||
});
|
||||
}
|
||||
|
||||
document.addEventListener('DOMContentLoaded',function(){
|
||||
var fileInput=document.getElementById('file');
|
||||
var drop=document.getElementById('drop');
|
||||
fileInput.addEventListener('change',function(e){
|
||||
if(e.target.files.length) readFile(e.target.files[0]);
|
||||
});
|
||||
drop.addEventListener('dragover',function(e){e.preventDefault();drop.classList.add('active')});
|
||||
drop.addEventListener('dragleave',function(){drop.classList.remove('active')});
|
||||
drop.addEventListener('drop',function(e){
|
||||
e.preventDefault();drop.classList.remove('active');
|
||||
if(e.dataTransfer.files.length) readFile(e.dataTransfer.files[0]);
|
||||
});
|
||||
|
||||
document.getElementById('use-sample').addEventListener('click',function(){
|
||||
// Fetch the sample CSV and feed it through the same pipeline
|
||||
fetch(A+'/samples/staffing_roster_sample.csv').then(function(r){return r.blob()}).then(function(blob){
|
||||
var file=new File([blob],'staffing_roster_sample.csv',{type:'text/csv'});
|
||||
readFile(file);
|
||||
}).catch(function(e){ alert('Sample unavailable: '+e.message); });
|
||||
});
|
||||
|
||||
document.getElementById('commit-btn').addEventListener('click',commit);
|
||||
document.getElementById('back-btn').addEventListener('click',function(){ setStep(2); });
|
||||
});
|
||||
</script>
|
||||
</body></html>
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"name": "mcp-server",
|
||||
"module": "index.ts",
|
||||
"type": "module",
|
||||
"private": true,
|
||||
"devDependencies": {
|
||||
"@types/bun": "latest"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"typescript": "^5"
|
||||
},
|
||||
"dependencies": {
|
||||
"@modelcontextprotocol/sdk": "^1.29.0"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,457 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en"><head>
|
||||
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>Lakehouse — Architecture & Reproduction</title>
|
||||
<style>
|
||||
*{margin:0;padding:0;box-sizing:border-box}
|
||||
body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;color:#b0b8c4;font-size:14px;line-height:1.55;-webkit-font-smoothing:antialiased}
|
||||
a{color:#58a6ff;text-decoration:none}
|
||||
a:hover{color:#79c0ff}
|
||||
|
||||
.bar{background:#0d1117;padding:0 24px;height:56px;border-bottom:1px solid #171d27;display:flex;justify-content:space-between;align-items:center;position:sticky;top:0;z-index:10}
|
||||
.bar h1{font-size:14px;font-weight:600;color:#e6edf3;letter-spacing:-0.2px}
|
||||
.bar nav{display:flex;gap:2px}
|
||||
.bar nav a{font-size:12px;color:#545d68;padding:6px 14px;border-radius:6px;transition:all 0.15s}
|
||||
.bar nav a:hover{color:#e6edf3;background:#161b22}
|
||||
.bar nav a.active{color:#e6edf3;background:#1c2333}
|
||||
.bar .rt{font-size:11px;color:#545d68}
|
||||
|
||||
.wrap{max-width:1040px;margin:0 auto;padding:28px 20px 60px}
|
||||
|
||||
.chapter{margin-bottom:48px}
|
||||
.chapter .num{color:#545d68;font-size:11px;font-weight:600;letter-spacing:1.6px;text-transform:uppercase;margin-bottom:6px}
|
||||
.chapter h2{color:#e6edf3;font-size:24px;font-weight:700;letter-spacing:-0.4px;margin-bottom:8px;line-height:1.2}
|
||||
.chapter .lede{color:#8b949e;font-size:14px;margin-bottom:18px;max-width:680px;line-height:1.6}
|
||||
.chapter h3{color:#e6edf3;font-size:16px;font-weight:600;margin:18px 0 6px}
|
||||
|
||||
.card{background:#0d1117;border:1px solid #171d27;border-radius:12px;padding:20px;margin-bottom:12px}
|
||||
.grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(220px,1fr));gap:10px}
|
||||
|
||||
.stat-lg{padding:18px 20px}
|
||||
.stat-lg .n{font-size:26px;font-weight:800;color:#e6edf3;letter-spacing:-1px;line-height:1}
|
||||
.stat-lg .l{font-size:10px;color:#545d68;text-transform:uppercase;letter-spacing:1.2px;margin-top:8px;font-weight:600}
|
||||
.stat-lg .sub{font-size:12px;color:#8b949e;margin-top:4px}
|
||||
|
||||
.accent-l{border-left:3px solid #2ea043}
|
||||
.accent-b{border-left:3px solid #1f6feb}
|
||||
.accent-a{border-left:3px solid #bc8cff}
|
||||
.accent-w{border-left:3px solid #d29922}
|
||||
.accent-r{border-left:3px solid #f85149}
|
||||
|
||||
.row{display:flex;justify-content:space-between;align-items:center;gap:12px;padding:10px 14px;background:#0d1117;border:1px solid #171d27;border-radius:8px;margin-bottom:6px;font-size:13px}
|
||||
.row:hover{border-color:#21262d}
|
||||
.row .title{color:#e6edf3;font-weight:500}
|
||||
.row .meta{color:#8b949e;font-size:11px;margin-top:2px;font-family:ui-monospace,Menlo,monospace}
|
||||
.row .val{color:#58a6ff;font-weight:600;white-space:nowrap;font-family:ui-monospace,Menlo,monospace}
|
||||
.row.pass{border-left:3px solid #2ea043}
|
||||
.row.fail{border-left:3px solid #f85149}
|
||||
|
||||
table.plain{width:100%;border-collapse:collapse;font-size:12px;margin-top:8px}
|
||||
table.plain th{text-align:left;padding:8px 12px;color:#545d68;font-weight:600;text-transform:uppercase;font-size:10px;letter-spacing:0.8px;border-bottom:1px solid #171d27}
|
||||
table.plain td{padding:8px 12px;border-bottom:1px solid #171d27;color:#c9d1d9;font-family:ui-monospace,Menlo,monospace}
|
||||
table.plain tr:hover td{background:#0d1117}
|
||||
|
||||
code{background:#161b22;color:#e6edf3;padding:2px 6px;border-radius:4px;font-family:ui-monospace,Menlo,monospace;font-size:12px}
|
||||
pre{background:#161b22;border:1px solid #171d27;border-radius:8px;padding:14px 16px;overflow-x:auto;font-family:ui-monospace,Menlo,monospace;font-size:12px;color:#c9d1d9;line-height:1.5;margin:8px 0}
|
||||
|
||||
.narr{color:#8b949e;font-size:13px;line-height:1.7;margin:10px 0;padding:10px 14px;border-left:2px solid #21262d}
|
||||
.narr strong{color:#c9d1d9;font-weight:600}
|
||||
|
||||
.ref{color:#545d68;font-size:11px;margin-top:6px;font-family:ui-monospace,Menlo,monospace}
|
||||
.ref strong{color:#79c0ff;font-weight:600}
|
||||
|
||||
.math{background:#0d1117;border:1px solid #171d27;border-radius:8px;padding:14px 16px;font-family:ui-monospace,Menlo,monospace;font-size:13px;color:#e6edf3;margin:8px 0}
|
||||
.math .c{color:#8b949e}
|
||||
|
||||
.footer{border-top:1px solid #171d27;padding:20px;text-align:center;color:#3d444d;font-size:11px}
|
||||
|
||||
.loading{color:#484f58;font-style:italic;padding:20px 0;text-align:center}
|
||||
.err{color:#f85149;font-size:12px;padding:10px}
|
||||
|
||||
@media(max-width:720px){
|
||||
.wrap{padding:20px 12px 40px}
|
||||
.chapter h2{font-size:20px}
|
||||
.bar nav{display:none}
|
||||
}
|
||||
</style></head>
|
||||
<body>
|
||||
|
||||
<div class="bar">
|
||||
<h1>Lakehouse — Architecture & Reproduction</h1>
|
||||
<nav>
|
||||
<a href=".">Dashboard</a>
|
||||
<a href="console">Walkthrough</a>
|
||||
<a href="proof" class="active">Architecture</a>
|
||||
<a href="spec">Spec</a>
|
||||
<a href="onboard">Onboard</a>
|
||||
<a href="alerts">Alerts</a>
|
||||
<a href="workspaces">Workspaces</a>
|
||||
</nav>
|
||||
<div class="rt" id="hdr-time">Running live tests…</div>
|
||||
</div>
|
||||
|
||||
<div class="wrap">
|
||||
|
||||
<div class="chapter">
|
||||
<div class="num">Chapter 1</div>
|
||||
<h2>Receipts, not promises</h2>
|
||||
<div class="lede">Every test below ran live against the real gateway when you loaded this page. Sub-100ms SQL on multi-million-row Parquet, hybrid search with playbook boost applied. No fixtures. If a test fails, you'll see ✗.</div>
|
||||
<div id="ch1-tests"><div class="loading">Running tests…</div></div>
|
||||
</div>
|
||||
|
||||
<div class="chapter">
|
||||
<div class="num">Chapter 2</div>
|
||||
<h2>Architecture — 13 crates, one object store, one local AI runtime</h2>
|
||||
<div class="lede">Request flows top to bottom. Every node is independently swappable. Every line is a real HTTP or gRPC hop that you can trace with <code>tcpdump</code>.</div>
|
||||
<div class="card accent-b">
|
||||
<pre> HTTP :3100 + gRPC :3101
|
||||
│
|
||||
┌───────▼───────┐
|
||||
│ gateway │ Rust · Axum · routing, CORS, auth, tools
|
||||
└───────┬───────┘
|
||||
┌────────────┬───────────┼───────────┬────────────┐
|
||||
│ │ │ │ │
|
||||
┌────▼───┐ ┌────▼───┐ ┌────▼───┐ ┌────▼───┐ ┌────▼───┐
|
||||
│catalog │ │ query │ │ vector │ │ ingest │ │aibridge│
|
||||
│ d │ │ d │ │ d │ │ d │ │ │
|
||||
└────┬───┘ └────┬───┘ └────┬───┘ └────┬───┘ └────┬───┘
|
||||
│ │ │ │ │
|
||||
└────────────┴───────────┼───────────┴────────────┘
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ object storage │ Parquet files (local / S3)
|
||||
└─────────────────┘
|
||||
▲
|
||||
│
|
||||
┌───────┴────────┐
|
||||
│ Python sidecar │ FastAPI → Ollama
|
||||
│ (aibridge) │ local models only
|
||||
└────────────────┘</pre>
|
||||
</div>
|
||||
<h3>Per-crate responsibility</h3>
|
||||
<table class="plain">
|
||||
<thead><tr><th>Crate</th><th>Role</th><th>Path</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>shared</td><td>Types, errors, Arrow helpers, PII detection, secrets provider</td><td>crates/shared/</td></tr>
|
||||
<tr><td>storaged</td><td>object_store I/O, BucketRegistry (multi-bucket), AppendLog, ErrorJournal</td><td>crates/storaged/</td></tr>
|
||||
<tr><td>catalogd</td><td>Metadata authority — manifests, views, tombstones, profiles, schema fingerprints</td><td>crates/catalogd/</td></tr>
|
||||
<tr><td>queryd</td><td>DataFusion SQL engine, MemTable cache, delta merge-on-read, compaction</td><td>crates/queryd/</td></tr>
|
||||
<tr><td>ingestd</td><td>CSV/JSON/PDF(+OCR)/Postgres/MySQL ingest, cron schedules, auto-PII</td><td>crates/ingestd/</td></tr>
|
||||
<tr><td>vectord</td><td>Embeddings as Parquet, HNSW, trial system, autotune agent, playbook_memory</td><td>crates/vectord/</td></tr>
|
||||
<tr><td>vectord-lance</td><td>Firewall crate — Lance 4.0 + Arrow 57 isolated from main Arrow 55</td><td>crates/vectord-lance/</td></tr>
|
||||
<tr><td>journald</td><td>Append-only mutation event log for time-travel & audit</td><td>crates/journald/</td></tr>
|
||||
<tr><td>aibridge</td><td>Rust↔Python sidecar, Ollama HTTP client, VRAM introspection</td><td>crates/aibridge/</td></tr>
|
||||
<tr><td>gateway</td><td>Axum HTTP :3100 + gRPC :3101, middleware, tools registry</td><td>crates/gateway/</td></tr>
|
||||
<tr><td>ui</td><td>Dioxus WASM internal developer UI</td><td>crates/ui/</td></tr>
|
||||
<tr><td>mcp-server</td><td>Bun TypeScript recruiter-facing app (this server)</td><td>mcp-server/</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<div class="ref"><strong>Source:</strong> git.agentview.dev/profit/lakehouse · <strong>ADRs:</strong> docs/DECISIONS.md (currently 20 records)</div>
|
||||
</div>
|
||||
|
||||
<div class="chapter">
|
||||
<div class="num">Chapter 3</div>
|
||||
<h2>Dual-agent recursive consensus loop</h2>
|
||||
<div class="lede">The system we use to execute staffing fills is a dual-agent recursive protocol. Two agents with distinct roles iterate against a shared log until one of three terminal states is reached. It is deterministic in structure, stochastic in content, and verifiable through the per-run log artifact.</div>
|
||||
<h3>Agents and protocol</h3>
|
||||
<div class="card accent-a">
|
||||
<pre> task in
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────────────────────────┐
|
||||
│ EXECUTOR (mistral:latest) │
|
||||
│ ──────────────────────────────────────────────────────── │
|
||||
│ input: task spec + shared log + seen-candidates ledger │
|
||||
│ output: one JSON action per turn │
|
||||
│ · {kind:"plan",steps:[…]} │
|
||||
│ · {kind:"tool_call",tool,args,rationale} │
|
||||
│ · {kind:"propose_done",fills:[N of N]} │
|
||||
└───────────┬───────────────────────────────┬───────────────┘
|
||||
│ tool_call │ propose_done
|
||||
▼ │
|
||||
┌──────────────────────────┐ │
|
||||
│ TOOL DISPATCH │ │
|
||||
│ hybrid_search / sql │ │
|
||||
│ (against live gateway) │ │
|
||||
└──────────┬───────────────┘ │
|
||||
│ result (trimmed, exclusions) │
|
||||
▼ ▼
|
||||
┌───────────────────────────────────────────────────────────┐
|
||||
│ REVIEWER (qwen2.5:latest) │
|
||||
│ ──────────────────────────────────────────────────────── │
|
||||
│ input: task spec + shared log (including tool result) │
|
||||
│ output: {kind:"critique",verdict:"continue|drift| │
|
||||
│ approve_done",notes} │
|
||||
└───────────┬───────────────────────────────────────────────┘
|
||||
│
|
||||
┌─────┴─────┐
|
||||
▼ ▼ ▼
|
||||
continue drift approve_done + propose_done ⟹ SEAL
|
||||
(next turn) (cap ≈ 3 →
|
||||
hard abort)
|
||||
</pre>
|
||||
</div>
|
||||
<div class="ref"><strong>Code:</strong> tests/multi-agent/agent.ts (protocol + prompts) · tests/multi-agent/orchestrator.ts (run loop) · tests/multi-agent/scenario.ts (5-event warehouse week)</div>
|
||||
|
||||
<h3>Why "dual" — role specialization</h3>
|
||||
<div class="narr">
|
||||
<strong>The executor is an optimist.</strong> Its job is to produce progress: pull candidates, verify SQL, propose consensus. It's instructed to be decisive.
|
||||
<br><br>
|
||||
<strong>The reviewer is a pessimist.</strong> Its job is to catch drift: proposals that don't match the task's geography, fill count, or role. It's authorized to stop the loop.
|
||||
<br><br>
|
||||
This adversarial separation is cheaper and more deterministic than asking a single model to self-critique. The reviewer has a hard rule: on the turn after a <code>propose_done</code>, it MUST emit either <code>approve_done</code> or <code>drift</code> — it cannot stall with <code>continue</code>.
|
||||
</div>
|
||||
|
||||
<h3>Why "parallel" — orchestrator can fan out</h3>
|
||||
<div class="narr">
|
||||
<strong>Independent pairs run concurrently.</strong> <code>tests/multi-agent/run_e2e_rated.ts</code> runs two task-specific agent pairs via <code>Promise.all</code>. Ollama serializes inference at the model level, so "parallel" is concurrent orchestration — but the substrate (gateway, queryd, vectord) handles concurrent requests cleanly. Verified in the scenario harness: two contracts sealing simultaneously.
|
||||
</div>
|
||||
|
||||
<h3>Why "recursive" — each seal feeds the next</h3>
|
||||
<div class="narr">
|
||||
<strong>Consensus does not end at the sealed playbook.</strong> Every sealed playbook is persisted to <code>playbook_memory</code> via <code>POST /vectors/playbook_memory/seed</code>. The next hybrid search for a semantically similar operation consults that memory via <code>compute_boost_for(query_embedding, top_k, base_weight)</code> and re-ranks the candidate pool. The system builds on itself turn over turn, playbook over playbook.
|
||||
</div>
|
||||
|
||||
<h3>Termination guarantees</h3>
|
||||
<div class="math">
|
||||
<span class="c">// three paths out, every run has one of these:</span><br>
|
||||
sealed = executor.propose_done ∧ reviewer.approve_done ∧ fills.count == target<br>
|
||||
abort = consecutive_tool_errors ≥ MAX_TOOL_ERRORS (3) <span class="c">// executor can't form a valid call</span><br>
|
||||
abort = consecutive_drifts ≥ MAX_CONSECUTIVE_DRIFTS (3) <span class="c">// reviewer keeps flagging</span><br>
|
||||
abort = turn > MAX_TURNS (12) <span class="c">// no consensus reached in window</span>
|
||||
</div>
|
||||
<div class="narr">Every abort dumps the full log to <code>tests/multi-agent/playbooks/<id>-FAILED.json</code> for forensic review. No consensus is ever implicit.</div>
|
||||
</div>
|
||||
|
||||
<div class="chapter">
|
||||
<div class="num">Chapter 4</div>
|
||||
<h2>Playbook memory — the compounding feedback loop</h2>
|
||||
<div class="lede">A CRM stores events. This system turns events into re-ranking signal. Every sealed playbook endorses specific (worker, city, state) tuples. Every failure penalizes them. Every similar future query inherits the signal through cosine similarity.</div>
|
||||
|
||||
<h3>Seed shape</h3>
|
||||
<div class="math">
|
||||
PlaybookEntry {<br>
|
||||
playbook_id, <span class="c">// pb-seed-<sha8></span><br>
|
||||
operation, <span class="c">// "fill: Welder x2 in Toledo, OH"</span><br>
|
||||
approach, context, <span class="c">// short canonical — long strings dilute embedding</span><br>
|
||||
timestamp, <span class="c">// RFC3339</span><br>
|
||||
endorsed_names[], <span class="c">// validated against workers_500k for city+state</span><br>
|
||||
city, state, <span class="c">// parsed from operation</span><br>
|
||||
embedding <span class="c">// 768-d nomic-embed-text of text shape</span><br>
|
||||
}
|
||||
</div>
|
||||
<div class="ref"><strong>Code:</strong> crates/vectord/src/playbook_memory.rs (PlaybookEntry, FailureRecord, PlaybookMemoryState)</div>
|
||||
|
||||
<h3>Boost math (positive + decay + negative)</h3>
|
||||
<div class="math">
|
||||
<span class="c">// For each playbook pb among top-K most cosine-similar:</span><br>
|
||||
<span class="c">// given query embedding qv, constant base_weight, n_workers = |pb.endorsed_names|</span><br>
|
||||
<br>
|
||||
similarity = cosine(qv, pb.embedding) <span class="c">// skip if ≤ 0.05</span><br>
|
||||
age_days = (now - pb.timestamp) / 86_400 seconds<br>
|
||||
decay = e<sup>-age_days / 30</sup> <span class="c">// half-life = 30 days</span><br>
|
||||
<br>
|
||||
<span class="c">// For each endorsed worker in pb:</span><br>
|
||||
key = (pb.city, pb.state, name)<br>
|
||||
fail_count = failures[key] <span class="c">// # times this worker was marked no-show for same geo</span><br>
|
||||
penalty = 0.5<sup>min(fail_count, 20)</sup><br>
|
||||
<br>
|
||||
per_worker = similarity × base_weight × decay × penalty / n_workers<br>
|
||||
boost[key] = min(boost[key] + per_worker, MAX_BOOST_PER_WORKER)<br>
|
||||
<br>
|
||||
<span class="c">// MAX_BOOST_PER_WORKER = 0.25 — cap stops one popular worker from always winning</span>
|
||||
</div>
|
||||
<div class="ref"><strong>Code:</strong> crates/vectord/src/playbook_memory.rs::compute_boost_for · constants: MAX_BOOST_PER_WORKER, DEFAULT_TOP_K_PLAYBOOKS, BOOST_HALF_LIFE_DAYS</div>
|
||||
|
||||
<h3>Application at query time</h3>
|
||||
<div class="math">
|
||||
<span class="c">// In /vectors/hybrid handler (crates/vectord/src/service.rs):</span><br>
|
||||
1. SQL filter narrows workers_500k to geo/role/availability<br>
|
||||
2. Vector index returns top_k × 5 candidates by cosine to question<br>
|
||||
3. compute_boost_for(qv, k=200) returns boost map<br>
|
||||
4. For each candidate: parse (name, city, state) from chunk, look up boost, add to score<br>
|
||||
5. Re-sort sources by boosted score<br>
|
||||
6. Truncate to requested top_k, return with playbook_boost and playbook_citations
|
||||
</div>
|
||||
<div class="narr">
|
||||
<strong>Why k=200.</strong> Direct measurement showed cosine similarity clusters in the 0.55-0.67 band across all playbooks regardless of geo (nomic-embed-text has narrow discrimination on this kind of structured operation text). A k of 25 silently missed geo-matched playbooks. k=200 is the measured floor for reliably catching compounding. Brute-force over 200 × 768-d is sub-ms even on this hardware.
|
||||
</div>
|
||||
<div class="ref"><strong>Evidence:</strong> Chicago Electrician compounding test 2026-04-20 — Carmen Green, Anna Patel, Fatima Wilson went from rank >5 / boost 0 / 0 citations (run 0, no seed) to rank 1/2/3 / boost +0.250 (capped) / 3 citations each (run 3, after 3 identical seeds). Each seed increments citations; total boost caps at 0.25/worker.</div>
|
||||
|
||||
<h3>Write-through to SQL</h3>
|
||||
<div class="narr">
|
||||
<strong>successful_playbooks_live</strong> is a DataFusion-queryable Parquet surface maintained by <code>POST /vectors/playbook_memory/persist_sql</code>. Every <code>/log</code> from the recruiter UI triggers seed → persist_sql. The in-memory store and the SQL surface stay synchronized (full snapshot on each persist, safe because memory is source of truth).
|
||||
</div>
|
||||
<div class="ref"><strong>Code:</strong> crates/vectord/src/playbook_memory.rs::persist_to_sql · catalog-registered under "successful_playbooks_live"</div>
|
||||
|
||||
<h3>Pattern discovery (Path 2 — meta-index)</h3>
|
||||
<div class="narr">
|
||||
<strong>Beyond "who was endorsed."</strong> <code>POST /vectors/playbook_memory/patterns</code> takes a query, finds top-K similar past playbooks, pulls each endorsed worker's full workers_500k profile, and aggregates shared traits: recurring certifications, skill frequencies, modal archetype, reliability distribution. Returns a <code>discovered_pattern</code> string showing operator-actionable signal the user didn't explicitly query for.
|
||||
</div>
|
||||
<div class="ref"><strong>Code:</strong> crates/vectord/src/playbook_memory.rs::discover_patterns · <strong>Surfaces:</strong> /vectors/playbook_memory/patterns endpoint, /intelligence/chat response, /intelligence/permit_contracts cards</div>
|
||||
</div>
|
||||
|
||||
<div class="chapter">
|
||||
<div class="num">Chapter 5</div>
|
||||
<h2>Key architectural choices — what was picked and why</h2>
|
||||
<div class="lede">Each choice is documented in <code>docs/DECISIONS.md</code> (Architecture Decision Records). If you dispute any of these, the ADR names the alternatives we rejected and the measurement that drove the call.</div>
|
||||
<div class="card">
|
||||
<div class="row accent-b">
|
||||
<div style="flex:1"><div class="title">ADR-001 · Object storage as source of truth</div><div class="meta">No traditional database. All data is Parquet on S3-compatible object storage. Eliminates DB operational overhead; every engine can read Parquet.</div></div>
|
||||
</div>
|
||||
<div class="row accent-b">
|
||||
<div style="flex:1"><div class="title">ADR-008 · Embeddings stored as Parquet, not a vector DB</div><div class="meta">Keeps all data in one portable format. No Pinecone/Weaviate/Qdrant lock-in. Trade-off: brute-force search up to ~100K; HNSW beyond.</div></div>
|
||||
</div>
|
||||
<div class="row accent-l">
|
||||
<div style="flex:1"><div class="title">ADR-012 · Append-only event journal — never destroy evidence</div><div class="meta">Every mutation is appended. Compliance, audit, AI-decision forensics. Impossible to retrofit; easy to add now.</div></div>
|
||||
</div>
|
||||
<div class="row accent-a">
|
||||
<div style="flex:1"><div class="title">ADR-015 · Tool registry before raw SQL for agents</div><div class="meta">Named, governed, audited actions for agents. Permission checks, rate limits, parameter validation. MCP-compatible.</div></div>
|
||||
</div>
|
||||
<div class="row accent-w">
|
||||
<div style="flex:1"><div class="title">ADR-019 · Hybrid Parquet+HNSW ⊕ Lance vector backend</div><div class="meta">Parquet+HNSW primary (2.55× faster search at 100K). Lance secondary for index-build speed (14× faster), random fetch (112× faster), append (structural). Per-profile <code>vector_backend: Parquet | Lance</code>.</div></div>
|
||||
</div>
|
||||
<div class="row accent-r">
|
||||
<div style="flex:1"><div class="title">ADR-020 · Idempotent register() with schema-fingerprint gate</div><div class="meta">Same (name, fingerprint) reuses manifest. Different fingerprint = 409 Conflict. Prevents silent duplicate manifests. Cleanup run collapsed 374 → 31 datasets.</div></div>
|
||||
</div>
|
||||
<div class="row accent-l">
|
||||
<div style="flex:1"><div class="title">Phase 19 design note · Statistical + semantic, not neural</div><div class="meta">Meta-index is cosine similarity + endorsement aggregation. No model training. Rebuildable from <code>successful_playbooks</code> alone. Neural re-ranker deferred to Phase 20+ only if statistical floor plateaus.</div></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="chapter">
|
||||
<div class="num">Chapter 6</div>
|
||||
<h2>Measured at scale, on this machine</h2>
|
||||
<div class="lede">Hardware: i9 + 128GB RAM + Nvidia A4000 16GB VRAM. Numbers below are from <em>this</em> running instance. Refresh the page and they'll recompute.</div>
|
||||
<div class="grid" id="ch6-scale"><div class="loading">Loading scale data…</div></div>
|
||||
<div id="ch6-recall" style="margin-top:10px"></div>
|
||||
</div>
|
||||
|
||||
<div class="chapter">
|
||||
<div class="num">Chapter 7</div>
|
||||
<h2>Verify or dispute — reproduce it yourself</h2>
|
||||
<div class="lede">Every claim below is a curl away from falsification.</div>
|
||||
<div class="card">
|
||||
<div class="narr"><strong>Health.</strong> Should return <code>lakehouse ok</code>.</div>
|
||||
<pre>curl http://localhost:3100/health</pre>
|
||||
<div class="narr"><strong>Any SQL on multi-million-row Parquet.</strong> Sub-100ms typical.</div>
|
||||
<pre>curl -s -X POST http://localhost:3100/query/sql \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"sql":"SELECT role, COUNT(*) FROM workers_500k WHERE state=\"IL\" GROUP BY role LIMIT 5"}'</pre>
|
||||
<div class="narr"><strong>Hybrid search with playbook boost.</strong> The whole Phase 19 feedback loop in one request.</div>
|
||||
<pre>curl -s -X POST http://localhost:3100/vectors/hybrid \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"index_name":"workers_500k_v1",
|
||||
"sql_filter":"role = '\''Forklift Operator'\'' AND city = '\''Chicago'\'' AND CAST(availability AS DOUBLE) > 0.5",
|
||||
"question":"reliable forklift operator",
|
||||
"top_k":5,"use_playbook_memory":true,"playbook_memory_k":200}'</pre>
|
||||
<div class="narr"><strong>Playbook memory stats.</strong> Count + endorsed names + sample.</div>
|
||||
<pre>curl http://localhost:3100/vectors/playbook_memory/stats</pre>
|
||||
<div class="narr"><strong>Pattern discovery.</strong> What do past similar fills have in common?</div>
|
||||
<pre>curl -s -X POST http://localhost:3100/vectors/playbook_memory/patterns \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"query":"Forklift Operator in Chicago, IL","top_k_playbooks":25,"min_trait_frequency":0.3}'</pre>
|
||||
<div class="narr"><strong>Run the dual-agent scenario yourself.</strong> All 5 events, real fills, real artifacts.</div>
|
||||
<pre>cd /home/profit/lakehouse
|
||||
bun run tests/multi-agent/scenario.ts
|
||||
# Output: tests/multi-agent/playbooks/scenario-<timestamp>/report.md</pre>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="chapter">
|
||||
<div class="num">Chapter 8</div>
|
||||
<h2>What we are <em>not</em> claiming</h2>
|
||||
<div class="lede">Every impressive-sounding number comes with a footnote. Here are the honest limits.</div>
|
||||
<div class="card">
|
||||
<div class="row accent-a"><div style="flex:1"><div class="title">workers_500k is synthetic.</div><div class="meta">Real client ATS export replaces this table. Schema is deliberately identical to a production ATS.</div></div></div>
|
||||
<div class="row accent-a"><div style="flex:1"><div class="title">candidates table has 1,000 rows.</div><div class="meta">Intentionally small for demo. call_log references higher candidate_ids that don't cross-reference — this is a dataset alignment issue, not a pipeline issue.</div></div></div>
|
||||
<div class="row accent-b"><div style="flex:1"><div class="title">Chicago permit data is real.</div><div class="meta">Pulled live from data.cityofchicago.org/resource/ydr8-5enu.json (Socrata API). Not synthetic. Not cached.</div></div></div>
|
||||
<div class="row accent-l"><div style="flex:1"><div class="title">Playbook memory is seeded from demo runs.</div><div class="meta">The pipeline that seeds it is identical to what a live recruiter would trigger via /log. Same code path.</div></div></div>
|
||||
<div class="row accent-w"><div style="flex:1"><div class="title">Local 7B models (mistral, qwen2.5) are imperfect.</div><div class="meta">They occasionally malform tool calls or drop fields. Multi-agent scenarios seal roughly 40-80% in one run. Larger models or constrained decoding would improve this. Not a substrate problem.</div></div></div>
|
||||
<div class="row accent-r"><div style="flex:1"><div class="title">No rate/margin awareness yet.</div><div class="meta">Worker pay expectations vs contract bill rates are not modeled. Flagged as a Phase 20 item; no architectural blocker.</div></div></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<div class="footer">Lakehouse · Architecture page regenerates tests on every load · <a href="console">walkthrough</a> · <a href=".">dashboard</a></div>
|
||||
|
||||
<script>
|
||||
var P=location.pathname.indexOf('/lakehouse')>=0?'/lakehouse':'';
|
||||
var A=location.origin+P;
|
||||
|
||||
function el(tag, cls, text){
|
||||
var e=document.createElement(tag);
|
||||
if(cls) e.className=cls;
|
||||
if(text!==undefined && text!==null) e.textContent=String(text);
|
||||
return e;
|
||||
}
|
||||
|
||||
function apiPost(path, body){
|
||||
return fetch(A+path,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(body||{})}).then(function(r){return r.json()});
|
||||
}
|
||||
|
||||
window.addEventListener('load',function(){
|
||||
loadLiveSections();
|
||||
});
|
||||
|
||||
function loadLiveSections(){
|
||||
apiPost('/proof.json',{}).then(function(r){
|
||||
var host1=document.getElementById('ch1-tests');host1.textContent='';
|
||||
var tests=r.tests||[];
|
||||
tests.forEach(function(t){
|
||||
var row=el('div','row '+(t.pass?'pass':'fail'));
|
||||
var left=document.createElement('div');left.style.flex='1';left.style.minWidth='0';
|
||||
var title=el('div','title',(t.pass?'✓ ':'✗ ')+(t.name||'(unnamed)'));
|
||||
var resultStr='';
|
||||
if(typeof t.result==='object' && t.result) resultStr=JSON.stringify(t.result);
|
||||
else if(t.result!==undefined && t.result!==null) resultStr=String(t.result);
|
||||
var meta=el('div','meta',resultStr);
|
||||
left.appendChild(title);left.appendChild(meta);
|
||||
row.appendChild(left);
|
||||
row.appendChild(el('div','val',(t.ms||0)+' ms'));
|
||||
host1.appendChild(row);
|
||||
});
|
||||
|
||||
var host6=document.getElementById('ch6-scale');host6.textContent='';
|
||||
var scale=r.scale||{};
|
||||
addStat(host6,(scale.total_rows||0).toLocaleString(),'Rows under management','across '+(scale.datasets||0)+' datasets','accent-b');
|
||||
addStat(host6,(scale.total_chunks||0).toLocaleString(),'Vector chunks indexed','across '+(scale.indexes||0)+' HNSW + Lance indexes','accent-a');
|
||||
var gpu=r.gpu||{};
|
||||
var vramStr=(gpu.vram_used_mib!==undefined?gpu.vram_used_mib+' / '+gpu.vram_total_mib+' MiB':'—');
|
||||
addStat(host6,vramStr,'GPU VRAM',(gpu.name||'A4000'),'accent-w');
|
||||
if(r.recall){
|
||||
var reco=r.recall||{};
|
||||
var hnswStr=(reco.hnsw!==undefined?(reco.hnsw*100).toFixed(0)+'%':'—');
|
||||
addStat(host6,hnswStr,'HNSW recall','measured · '+(reco.note||''),'accent-l');
|
||||
}
|
||||
|
||||
var host6r=document.getElementById('ch6-recall');host6r.textContent='';
|
||||
if(r.lance_10m){
|
||||
var narr=el('div','narr');
|
||||
narr.appendChild(el('strong',null,'Scale headroom: '));
|
||||
narr.appendChild(document.createTextNode(
|
||||
'Lance backend tested at '+r.lance_10m.vectors.toLocaleString()+' vectors, '
|
||||
+r.lance_10m.disk_gb+' GB on disk, '+r.lance_10m.search_p50_ms+'ms p50. '
|
||||
+(r.lance_10m.note||'')
|
||||
));
|
||||
host6r.appendChild(narr);
|
||||
}
|
||||
|
||||
document.getElementById('hdr-time').textContent='Generated · '+(r.generated?new Date(r.generated).toLocaleTimeString():'live');
|
||||
}).catch(function(e){
|
||||
var host1=document.getElementById('ch1-tests');host1.textContent='';
|
||||
host1.appendChild(el('div','err','Live tests failed: '+(e.message||e)+'. Server may be offline.'));
|
||||
});
|
||||
}
|
||||
|
||||
function addStat(host,n,l,sub,cls){
|
||||
var d=el('div','card stat-lg '+(cls||''));
|
||||
d.appendChild(el('div','n',n));
|
||||
d.appendChild(el('div','l',l));
|
||||
d.appendChild(el('div','sub',sub||''));
|
||||
host.appendChild(d);
|
||||
}
|
||||
</script>
|
||||
</body></html>
|
||||
@@ -0,0 +1,26 @@
|
||||
worker_id,name,role,city,state,email,phone,skills,certifications,availability,reliability,archetype
|
||||
W-1001,Sarah Johnson,Forklift Operator,Chicago,IL,sarah.johnson@example.com,(312) 555-0101,pallet jack|hazmat|loading dock,OSHA-10|Forklift,0.92,0.88,reliable
|
||||
W-1002,Michael Chen,Welder,Toledo,OH,m.chen@example.com,(419) 555-0234,TIG|MIG|blueprint reading|grinder,OSHA-10|AWS D1.1,0.71,0.94,specialist
|
||||
W-1003,Maria Rodriguez,Warehouse Associate,Chicago,IL,maria.r@example.com,(312) 555-0311,inventory|RF scanner|pick-to-light,OSHA-10,0.85,0.80,reliable
|
||||
W-1004,David Park,Machine Operator,Detroit,MI,dpark@example.com,(313) 555-0412,CNC|gauge R&R|lean manufacturing,OSHA-10|First Aid,0.60,0.91,specialist
|
||||
W-1005,Jennifer Lopez,Loader,Chicago,IL,j.lopez@example.com,(312) 555-0501,loading dock|team lead|cold storage,OSHA-10|Forklift,0.95,0.76,communicator
|
||||
W-1006,Robert Williams,Maintenance Tech,Milwaukee,WI,rwilliams@example.com,(414) 555-0612,electrical|PLC|hydraulics|CMMS,OSHA-10|LOTO,0.45,0.93,specialist
|
||||
W-1007,Amanda Taylor,Quality Tech,Toledo,OH,a.taylor@example.com,(419) 555-0701,ISO 9001|calibration|root cause analysis,OSHA-10|Six Sigma Green,0.80,0.87,leader
|
||||
W-1008,Carlos Mendoza,Electrician,Chicago,IL,c.mendoza@example.com,(312) 555-0808,conduit|motor controls|troubleshooting,OSHA-30|NEC,0.88,0.96,specialist
|
||||
W-1009,Kim Nguyen,Forklift Operator,Detroit,MI,k.nguyen@example.com,(313) 555-0915,pallet jack|shipping|team lead,OSHA-10|Forklift,0.73,0.82,flexible
|
||||
W-1010,James O'Brien,Welder,Milwaukee,WI,jobrien@example.com,(414) 555-1021,TIG|pipe welding|confined space,OSHA-10|AWS D1.1,0.90,0.79,reliable
|
||||
W-1011,Priya Patel,Production Worker,Chicago,IL,priya.p@example.com,(312) 555-1118,line work|first article|labeling,OSHA-10,0.66,0.85,reliable
|
||||
W-1012,Thomas Anderson,Material Handler,Toledo,OH,tanderson@example.com,(419) 555-1225,RF scanner|pallet jack|receiving,OSHA-10|Forklift,0.82,0.77,flexible
|
||||
W-1013,Lisa Nakamura,Warehouse Associate,Detroit,MI,l.nakamura@example.com,(313) 555-1312,inventory|Excel|packaging,OSHA-10,0.77,0.89,communicator
|
||||
W-1014,Brandon Moore,Machine Operator,Milwaukee,WI,b.moore@example.com,(414) 555-1405,CNC|SPC|conveyor ops,OSHA-10,0.56,0.94,specialist
|
||||
W-1015,Emily Zhang,Assembler,Chicago,IL,emily.z@example.com,(312) 555-1502,assembly|gauge R&R|line lead,OSHA-10|Six Sigma Yellow,0.91,0.81,leader
|
||||
W-1016,Marcus Johnson,Forklift Operator,Toledo,OH,mjohnson@example.com,(419) 555-1609,pallet jack|hazmat|overhead crane,OSHA-10|Forklift|Hazmat,0.68,0.88,reliable
|
||||
W-1017,Anita Brooks,Line Lead,Detroit,MI,a.brooks@example.com,(313) 555-1715,team lead|training|SPC,OSHA-30|Six Sigma Green,0.52,0.97,leader
|
||||
W-1018,Dmitri Volkov,Welder,Chicago,IL,d.volkov@example.com,(312) 555-1820,TIG|MIG|blueprint reading,OSHA-10|AWS D1.1|Confined Space,0.84,0.83,specialist
|
||||
W-1019,Rachel Kim,Loader,Milwaukee,WI,r.kim@example.com,(414) 555-1911,loading dock|first aid|bilingual,OSHA-10|First Aid,0.79,0.84,communicator
|
||||
W-1020,Samuel Park,Maintenance Tech,Toledo,OH,spark@example.com,(419) 555-2012,electrical|PLC|troubleshooting|CMMS,OSHA-10|LOTO,0.70,0.92,specialist
|
||||
W-1021,Jordan Williams,Quality Tech,Detroit,MI,j.williams@example.com,(313) 555-2117,ISO 9001|calibration|SPC,OSHA-10|Six Sigma Green,0.83,0.86,leader
|
||||
W-1022,Natalia Soto,Warehouse Associate,Chicago,IL,n.soto@example.com,(312) 555-2221,RF scanner|inventory|team lead,OSHA-10,0.89,0.90,communicator
|
||||
W-1023,Henry Chen,Machine Operator,Milwaukee,WI,h.chen@example.com,(414) 555-2318,CNC|conveyor ops|root cause,OSHA-10,0.64,0.93,specialist
|
||||
W-1024,Ava Martinez,Forklift Operator,Toledo,OH,a.martinez@example.com,(419) 555-2411,pallet jack|bilingual|cold storage,OSHA-10|Forklift,0.87,0.80,flexible
|
||||
W-1025,Tyler Rodriguez,Production Worker,Chicago,IL,t.rodriguez@example.com,(312) 555-2515,line work|packaging|quality inspection,OSHA-10,0.74,0.86,reliable
|
||||
|
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,501 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en"><head>
|
||||
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>Lakehouse — Technical Specification</title>
|
||||
<style>
|
||||
*{margin:0;padding:0;box-sizing:border-box}
|
||||
body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;color:#b0b8c4;font-size:14px;line-height:1.6;-webkit-font-smoothing:antialiased}
|
||||
a{color:#58a6ff;text-decoration:none}
|
||||
a:hover{color:#79c0ff}
|
||||
|
||||
.bar{background:#0d1117;padding:0 24px;height:56px;border-bottom:1px solid #171d27;display:flex;justify-content:space-between;align-items:center;position:sticky;top:0;z-index:10}
|
||||
.bar h1{font-size:14px;font-weight:600;color:#e6edf3;letter-spacing:-0.2px}
|
||||
.bar nav{display:flex;gap:2px}
|
||||
.bar nav a{font-size:12px;color:#545d68;padding:6px 14px;border-radius:6px;transition:all 0.15s}
|
||||
.bar nav a:hover{color:#e6edf3;background:#161b22}
|
||||
.bar nav a.active{color:#e6edf3;background:#1c2333}
|
||||
.bar .rt{font-size:11px;color:#545d68}
|
||||
|
||||
.layout{display:grid;grid-template-columns:220px 1fr;gap:0;max-width:1200px;margin:0 auto}
|
||||
.toc{position:sticky;top:72px;align-self:start;padding:24px 12px;border-right:1px solid #171d27;height:calc(100vh - 72px);overflow-y:auto}
|
||||
.toc .hdr{color:#545d68;font-size:10px;text-transform:uppercase;letter-spacing:1.4px;font-weight:600;margin-bottom:10px;padding-left:8px}
|
||||
.toc a{display:block;color:#8b949e;font-size:12px;padding:6px 10px;border-radius:6px;margin-bottom:2px;line-height:1.4}
|
||||
.toc a:hover{color:#e6edf3;background:#161b22}
|
||||
|
||||
.wrap{padding:28px 24px 60px;min-width:0}
|
||||
|
||||
.chapter{margin-bottom:48px;scroll-margin-top:72px}
|
||||
.chapter .num{color:#545d68;font-size:11px;font-weight:600;letter-spacing:1.6px;text-transform:uppercase;margin-bottom:6px}
|
||||
.chapter h2{color:#e6edf3;font-size:24px;font-weight:700;letter-spacing:-0.4px;margin-bottom:8px;line-height:1.2}
|
||||
.chapter h3{color:#e6edf3;font-size:16px;font-weight:600;margin:22px 0 8px}
|
||||
.chapter h4{color:#c9d1d9;font-size:14px;font-weight:600;margin:16px 0 6px}
|
||||
.chapter .lede{color:#8b949e;font-size:14px;margin-bottom:18px;max-width:760px;line-height:1.7}
|
||||
.chapter p{color:#b0b8c4;margin-bottom:12px;max-width:800px}
|
||||
.chapter ul{color:#b0b8c4;margin:8px 0 16px 24px}
|
||||
.chapter li{margin-bottom:6px;line-height:1.7}
|
||||
.chapter strong{color:#e6edf3;font-weight:600}
|
||||
|
||||
.card{background:#0d1117;border:1px solid #171d27;border-radius:12px;padding:20px;margin:12px 0}
|
||||
.accent-l{border-left:3px solid #2ea043}
|
||||
.accent-b{border-left:3px solid #1f6feb}
|
||||
.accent-a{border-left:3px solid #bc8cff}
|
||||
.accent-w{border-left:3px solid #d29922}
|
||||
.accent-r{border-left:3px solid #f85149}
|
||||
|
||||
code{background:#161b22;color:#e6edf3;padding:2px 6px;border-radius:4px;font-family:ui-monospace,Menlo,monospace;font-size:12px}
|
||||
pre{background:#161b22;border:1px solid #171d27;border-radius:8px;padding:14px 16px;overflow-x:auto;font-family:ui-monospace,Menlo,monospace;font-size:12px;color:#c9d1d9;line-height:1.5;margin:8px 0}
|
||||
|
||||
table.plain{width:100%;border-collapse:collapse;font-size:12px;margin:10px 0}
|
||||
table.plain th{text-align:left;padding:8px 12px;color:#545d68;font-weight:600;text-transform:uppercase;font-size:10px;letter-spacing:0.8px;border-bottom:1px solid #171d27;background:#0d1117}
|
||||
table.plain td{padding:8px 12px;border-bottom:1px solid #171d27;color:#c9d1d9;vertical-align:top}
|
||||
table.plain td.mono{font-family:ui-monospace,Menlo,monospace}
|
||||
table.plain tr:hover td{background:#0d1117}
|
||||
|
||||
.narr{color:#8b949e;font-size:13px;line-height:1.7;margin:10px 0;padding:10px 14px;border-left:2px solid #21262d}
|
||||
.narr strong{color:#c9d1d9}
|
||||
|
||||
.ref{color:#545d68;font-size:11px;margin-top:6px;font-family:ui-monospace,Menlo,monospace}
|
||||
.ref strong{color:#79c0ff;font-weight:600}
|
||||
|
||||
.step{display:flex;gap:14px;margin-bottom:14px;padding:12px 16px;background:#0d1117;border:1px solid #171d27;border-radius:8px}
|
||||
.step .n{color:#58a6ff;font-weight:700;font-size:18px;flex-shrink:0;min-width:30px}
|
||||
.step .body{color:#b0b8c4;font-size:13px;line-height:1.7}
|
||||
.step .body strong{color:#e6edf3}
|
||||
|
||||
.footer{border-top:1px solid #171d27;padding:20px;text-align:center;color:#3d444d;font-size:11px}
|
||||
|
||||
@media(max-width:900px){
|
||||
.layout{grid-template-columns:1fr}
|
||||
.toc{display:none}
|
||||
.wrap{padding:20px 14px 40px}
|
||||
.bar nav{display:none}
|
||||
}
|
||||
</style></head>
|
||||
<body>
|
||||
|
||||
<div class="bar">
|
||||
<h1>Lakehouse — Technical Specification</h1>
|
||||
<nav>
|
||||
<a href=".">Dashboard</a>
|
||||
<a href="console">Walkthrough</a>
|
||||
<a href="proof">Architecture</a>
|
||||
<a href="spec" class="active">Spec</a>
|
||||
<a href="onboard">Onboard</a>
|
||||
<a href="alerts">Alerts</a>
|
||||
<a href="workspaces">Workspaces</a>
|
||||
</nav>
|
||||
<div class="rt">v1 · 2026-04-20</div>
|
||||
</div>
|
||||
|
||||
<div class="layout">
|
||||
<aside class="toc">
|
||||
<div class="hdr">Contents</div>
|
||||
<a href="#ch1">1. Repository layout</a>
|
||||
<a href="#ch2">2. Data ingest pipeline</a>
|
||||
<a href="#ch3">3. Measurement & indexing</a>
|
||||
<a href="#ch4">4. Contract inference</a>
|
||||
<a href="#ch5">5. What a CRM can't do</a>
|
||||
<a href="#ch6">6. How it gets better over time</a>
|
||||
<a href="#ch7">7. Scale story — 20 staffers, 300 contracts, a surge</a>
|
||||
<a href="#ch8">8. Error surfaces & recovery</a>
|
||||
<a href="#ch9">9. Per-staffer context</a>
|
||||
<a href="#ch10">10. A day in the life</a>
|
||||
<a href="#ch11">11. Known limits & non-goals</a>
|
||||
</aside>
|
||||
<div class="wrap">
|
||||
|
||||
<!-- ═══ 1. REPO LAYOUT ═══ -->
|
||||
<div class="chapter" id="ch1">
|
||||
<div class="num">Chapter 1</div>
|
||||
<h2>Repository layout</h2>
|
||||
<div class="lede">What lives where. Every folder below has a single, bounded responsibility. A maintainer reading this should know — in under ten minutes — which crate owns a failing behavior.</div>
|
||||
<table class="plain">
|
||||
<thead><tr><th>Path</th><th>Owns</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td class="mono">crates/shared/</td><td>Types, errors, Arrow helpers, schema fingerprints, PII detection, secrets provider. Every other crate depends on this.</td></tr>
|
||||
<tr><td class="mono">crates/storaged/</td><td>Raw object I/O. <code>BucketRegistry</code> (multi-bucket, rescue-aware), <code>AppendLog</code> (write-once batched append), <code>ErrorJournal</code> (bucket op failures). ADR-017 (federation), ADR-018 (append pattern).</td></tr>
|
||||
<tr><td class="mono">crates/catalogd/</td><td>Metadata authority. Dataset manifests, schema fingerprints (ADR-020), tombstones (soft delete), AI-safe views, model profiles (Phase 17). In-memory index persisted as Parquet on storage.</td></tr>
|
||||
<tr><td class="mono">crates/queryd/</td><td>SQL engine. DataFusion over Parquet + MemTable cache + delta merge-on-read + compaction. Registers every bucket as an object_store so SQL can join across them.</td></tr>
|
||||
<tr><td class="mono">crates/ingestd/</td><td>Data on-ramp. CSV / JSON / PDF (+OCR via Tesseract) / Postgres streaming / MySQL streaming / inbox watcher / cron schedules. Every ingest path auto-tags PII (emails, phones, SSNs, addresses), records lineage, and marks embeddings stale.</td></tr>
|
||||
<tr><td class="mono">crates/vectord/</td><td>The vector + learning surface. Embeddings stored as Parquet (ADR-008), HNSW index (Phase 15), trial system (autotune), promotion registry (Phase 16), playbook_memory (Phase 19). Core feedback loop lives here.</td></tr>
|
||||
<tr><td class="mono">crates/vectord-lance/</td><td>Firewall crate. Lance 4.0 + Arrow 57, isolated from the main Arrow-55 workspace. Provides secondary vector backend for large-scale, random-access, and append-heavy workloads (ADR-019).</td></tr>
|
||||
<tr><td class="mono">crates/journald/</td><td>Append-only mutation event log (ADR-012). Every insert/update/delete writes here — who, when, what, old/new value. Never mutated. Foundation for time-travel + compliance audit.</td></tr>
|
||||
<tr><td class="mono">crates/aibridge/</td><td>Rust ↔ Python sidecar. HTTP client over FastAPI wrapper around Ollama. VRAM introspection via nvidia-smi. All LLM calls (embed, generate, rerank) flow through here.</td></tr>
|
||||
<tr><td class="mono">crates/gateway/</td><td>Axum HTTP (:3100) + gRPC (:3101). Auth middleware, tools registry (Phase 12 — governed actions), CORS. Every external request enters here.</td></tr>
|
||||
<tr><td class="mono">crates/ui/</td><td>Dioxus WASM developer UI. Internal tool. Not exposed externally.</td></tr>
|
||||
<tr><td class="mono">mcp-server/</td><td>Bun/TypeScript recruiter-facing app. Serves <code>devop.live/lakehouse</code>. Routes: <code>/search /match /log /log_failure /clients/:c/blacklist /intelligence/* /memory/query /models/matrix /system/summary</code>. Observer sibling at <code>observer.ts</code> with HTTP listener on :3800 for scenario event ingest. Proxies to the Rust gateway for heavy work.</td></tr>
|
||||
<tr><td class="mono">tests/multi-agent/</td><td>Dual-agent scenario harness + memory stack. <code>agent.ts</code> (prompts, continuation + tree-split primitives, cloud routing), <code>orchestrator.ts</code>, <code>scenario.ts</code> (contracts + staffer + tool_level), <code>kb.ts</code> (KB indexing, competence scoring, neighbor retrieval), <code>normalize.ts</code> (input normalizer — structured / regex / LLM), <code>memory_query.ts</code> (unified /memory/query), <code>gen_scenarios.ts</code> + <code>gen_staffer_demo.ts</code> (corpus generators), <code>run_e2e_rated.ts</code>, <code>chain_of_custody.ts</code>. Unit tests colocated (<code>kb.test.ts</code>, <code>normalize.test.ts</code>).</td></tr>
|
||||
<tr><td class="mono">config/</td><td><code>models.json</code> — authoritative 5-tier model matrix (T1 hot local / T2 review local / T3 overview cloud / T4 strategic / T5 gatekeeper). Per-tier context_window + context_budget + overflow_policy. Read at runtime by scenario.ts; hot-swap friendly.</td></tr>
|
||||
<tr><td class="mono">docs/</td><td><code>PRD.md</code>, <code>PHASES.md</code>, <code>DECISIONS.md</code> (20 ADRs). Every significant architectural choice has an ADR with the alternatives that were rejected and why.</td></tr>
|
||||
<tr><td class="mono">data/</td><td>Default local object store. Parquet files per dataset, append-log batches, HNSW trial journals, promotion registries, <code>_playbook_memory/state.json</code> (now with retirement fields — Phase 25), catalog manifests. Plus four learning-loop directories: <code>_kb/</code> (signatures, outcomes, recommendations, error_corrections, config_snapshots, staffers), <code>_playbook_lessons/</code> (T3 cross-day lessons archived per run), <code>_observer/ops.jsonl</code> (append journal, durable scenario outcome stream), <code>_chunk_cache/</code> (spec'd for Phase 21 Rust port). Rebuildable from repo + this dir alone.</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<!-- ═══ 2. INGEST PIPELINE ═══ -->
|
||||
<div class="chapter" id="ch2">
|
||||
<div class="num">Chapter 2</div>
|
||||
<h2>Data ingest pipeline</h2>
|
||||
<div class="lede">How staffing data gets into the system — whether from a CSV drop, an ATS export, a Postgres replica, or a PDF resume. Every path ends at the same place: a registered dataset with known schema, known lineage, known sensitivity.</div>
|
||||
|
||||
<div class="step"><div class="n">1</div><div class="body"><strong>Source arrives.</strong> Four shapes: (a) file upload via <code>POST /ingest/file</code>, (b) inbox watcher (drops in <code>./inbox/</code> → auto-ingested in under 15s), (c) Postgres or MySQL streaming connector (<code>POST /ingest/db</code> with DSN), (d) scheduled ingest via <code>ingestd::schedule</code> with cron.</div></div>
|
||||
|
||||
<div class="step"><div class="n">2</div><div class="body"><strong>Parse + normalize.</strong> CSV parser infers types per column; defaults to <code>String</code> on ambiguity (ADR-010 — better to ingest everything than reject on type mismatch). JSON parser flattens nested objects. PDF extractor uses <code>lopdf</code> first; falls back to Tesseract OCR for scanned/image PDFs. Output is always an Arrow <code>RecordBatch</code>.</div></div>
|
||||
|
||||
<div class="step"><div class="n">3</div><div class="body"><strong>Auto-detect PII.</strong> <code>shared::pii</code> scans column values and names. Identifies emails, phone numbers, SSNs, salaries, street addresses, medical terms. Tags columns with <code>sensitivity: PII | PHI | Financial | Internal | Public</code> (Phase 10 catalog v2).</div></div>
|
||||
|
||||
<div class="step"><div class="n">4</div><div class="body"><strong>Deduplicate by content hash.</strong> Every uploaded file's SHA-256 is checked against the catalog's seen-hash log. Re-ingesting the same file is a no-op (ADR invariant #5).</div></div>
|
||||
|
||||
<div class="step"><div class="n">5</div><div class="body"><strong>Write Parquet to object storage.</strong> <code>arrow_helpers::record_batch_to_parquet</code> → <code>storaged::ops::put</code> → file lands under <code>data/datasets/<name>.parquet</code> (or bucket-scoped via <code>BucketRegistry</code>). Schema fingerprint computed.</div></div>
|
||||
|
||||
<div class="step"><div class="n">6</div><div class="body"><strong>Register in catalog.</strong> <code>catalogd::Registry::register(name, fingerprint, objects)</code> — idempotent on (name, fingerprint). Same name + same fingerprint = reuse manifest, bump updated_at. Same name + different fingerprint = <code>409 Conflict</code> (ADR-020 — prevents silent schema drift). New name = create new manifest with owner, lineage, freshness SLA, column metadata, PII tags.</div></div>
|
||||
|
||||
<div class="step"><div class="n">7</div><div class="body"><strong>Mark embeddings stale.</strong> If the dataset already has a vector index, the new rows mean that index is now behind. <code>Registry::mark_embeddings_stale</code> flips a flag; <code>POST /vectors/refresh/<dataset></code> runs an incremental re-embed (only new rows, not the whole corpus).</div></div>
|
||||
|
||||
<div class="step"><div class="n">8</div><div class="body"><strong>Queryable immediately.</strong> <code>queryd::context</code> picks up the new manifest on next query. Hot-cache warms on first hit. Delta merge-on-read means updates land without rewriting the base Parquet.</div></div>
|
||||
|
||||
<div class="ref"><strong>Code:</strong> crates/ingestd/src/{service.rs, csv.rs, json.rs, pdf.rs, pg_stream.rs, my_stream.rs, schedule.rs}</div>
|
||||
</div>
|
||||
|
||||
<!-- ═══ 3. MEASUREMENT & INDEXING ═══ -->
|
||||
<div class="chapter" id="ch3">
|
||||
<div class="num">Chapter 3</div>
|
||||
<h2>Measurement & indexing</h2>
|
||||
<div class="lede">Once data is in, the system describes it rigorously and builds fast-access indexes over the parts that will be queried. Every measurement is deterministic, versioned, and visible via HTTP.</div>
|
||||
|
||||
<h3>What gets measured per dataset</h3>
|
||||
<ul>
|
||||
<li><strong>Row count</strong> (from parquet footer, not a SELECT COUNT). O(1).</li>
|
||||
<li><strong>Schema fingerprint</strong> — SHA-256 over (column_name, type, nullability, sort) tuples. Drives ADR-020 idempotent register.</li>
|
||||
<li><strong>Owner / sensitivity / freshness SLA</strong> — catalog v2 metadata. PII auto-detected; owner assigned on ingest.</li>
|
||||
<li><strong>Lineage</strong> — source_system → ingest_job → dataset. Who put this here, when, from what.</li>
|
||||
<li><strong>Last embedded at</strong> — when the vector index covering this dataset was last refreshed. Drives stale-detection.</li>
|
||||
</ul>
|
||||
|
||||
<h3>How vector indexes are built</h3>
|
||||
<p>Two backends, chosen per profile (ADR-019):</p>
|
||||
<table class="plain">
|
||||
<thead><tr><th></th><th>HNSW over Parquet (primary)</th><th>Lance (secondary)</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Storage</td><td>Embeddings as Parquet columns (<code>doc_id, chunk_text, vector</code>)</td><td>Lance native dataset</td></tr>
|
||||
<tr><td>Index</td><td>HNSW in RAM, serialized sidecar</td><td>IVF_PQ on disk</td></tr>
|
||||
<tr><td>Build time (100K × 768d)</td><td>~230s</td><td>~16s (14× faster)</td></tr>
|
||||
<tr><td>Search p50 (100K)</td><td>~873μs</td><td>~7.4ms at recall 1.0</td></tr>
|
||||
<tr><td>Append</td><td>Rewrite required</td><td>Structural (0.08s for 100 rows)</td></tr>
|
||||
<tr><td>Random fetch by doc_id</td><td>Full scan</td><td>~311μs (112× faster)</td></tr>
|
||||
<tr><td>RAM ceiling</td><td>~5M vectors</td><td>Scales past RAM — disk-resident</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h3>Autotune</h3>
|
||||
<p>The <code>vectord::agent</code> background task runs continuously. Per index, it proposes HNSW configurations (<code>ef_construction × ef_search</code>), executes a trial against a stored eval set, journals the result as JSONL, and — if recall beats the min_recall gate (0.9) and latency wins the Pareto test — promotes the new config atomically via <code>promotion_registry</code>. No downtime. Rollback in milliseconds.</p>
|
||||
|
||||
<h3>Per-profile / per-staffer indexing</h3>
|
||||
<p>Model profiles (Phase 17) are not routing strings — they are named scopes. Each profile has <code>bound_datasets[]</code>, <code>hnsw_config</code>, <code>vector_backend</code>, and <code>bucket</code>. When a staffer activates a profile:</p>
|
||||
<ul>
|
||||
<li>EmbeddingCache warms for bound indexes only</li>
|
||||
<li>HNSW is rebuilt with the profile's config (if different from current)</li>
|
||||
<li>Search via <code>POST /vectors/profile/<id>/search</code> rejects out-of-scope queries with 403 + list of allowed bindings</li>
|
||||
<li>Ollama swaps to the profile's model via <code>keep_alive=0</code>; only one model in VRAM at a time</li>
|
||||
</ul>
|
||||
|
||||
<h3>Model matrix (Phase 20)</h3>
|
||||
<p>Five tiers declared in <code>config/models.json</code>. Each call site picks the tier appropriate to its purpose — hot-path JSON emitters get fast local, overview/strategic/gatekeeper decisions get thinking models on cloud. Every tier carries <code>context_window</code>, <code>context_budget</code>, and <code>overflow_policy</code>.</p>
|
||||
<table class="plain">
|
||||
<thead><tr><th>Tier</th><th>Purpose</th><th>Primary model</th><th>Frequency</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>T1 hot</td><td>Per tool call — SQL gen, hybrid_search, propose_done</td><td><code>qwen3.5:latest</code> local, <code>think:false</code></td><td>50-200/scenario</td></tr>
|
||||
<tr><td>T2 review</td><td>Per-step consensus, drift flagging</td><td><code>qwen3:latest</code> local, <code>think:false</code></td><td>5-14/event</td></tr>
|
||||
<tr><td>T3 overview</td><td>Mid-day checkpoints + cross-day lesson distill</td><td><code>gpt-oss:120b</code> Ollama Cloud, thinking on</td><td>1-3/scenario</td></tr>
|
||||
<tr><td>T4 strategic</td><td>Pattern re-ranking, weekly gap audit</td><td><code>qwen3.5:397b</code> cloud</td><td>1-10/day</td></tr>
|
||||
<tr><td>T5 gatekeeper</td><td>Schema migrations, autotune config changes</td><td><code>kimi-k2-thinking</code> cloud, audit-logged</td><td>1-5/day</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p><strong>Key mechanical finding (2026-04-21):</strong> qwen3.5 and qwen3 are <em>thinking</em> models — they burn ~650 tokens of hidden reasoning before emitting the visible response. For hot-path JSON emitters this meant 400-token budgets returned empty strings. Fix: <code>think: false</code> plumbed through sidecar's <code>/generate</code> endpoint; hot path disables thinking (structure matters more than reasoning depth), overseer tiers keep it on. Mistral was dropped entirely after a 0/14 fill rate on complex scenarios (decoder-level malformed-JSON bug, not a prompt issue).</p>
|
||||
<p><strong>Continuation primitive (Phase 21):</strong> <code>generateContinuable()</code> handles output-overflow without <code>max_tokens</code> tourniquets — empty response → geometric backoff retry; truncated-JSON → continue with partial as scratchpad. <code>generateTreeSplit()</code> handles input-overflow via map-reduce with running scratchpad. Both respect <code>assertContextBudget()</code> so silent truncation can't happen.</p>
|
||||
|
||||
<h3>Per-staffer tool_level (Phase 23)</h3>
|
||||
<p>Scenarios can be scoped to a specific coordinator (<code>staffer: {id, name, tenure_months, role, tool_level}</code>). <code>tool_level</code> controls which tiers are available:</p>
|
||||
<ul>
|
||||
<li><code>full</code> — T1/T2 local, T3 cloud, cloud rescue on failure</li>
|
||||
<li><code>local</code> — T1/T2/T3 all local (gpt-oss:20b as overseer)</li>
|
||||
<li><code>basic</code> — kimi-k2.5 cloud executor + local reviewer + local T3, no rescue</li>
|
||||
<li><code>minimal</code> — kimi-k2.5 cloud executor, no T3, no rescue. Playbook inheritance is the only signal.</li>
|
||||
</ul>
|
||||
<p>Measured 2026-04-21 on a 36-run demo (4 staffers × 3 contracts × 3 rounds): James Park (mid, local tools) ranked first at 92.9% fill and 36.8 cites/run, Maria Chen (senior, full tools) second at 81.0%. Cloud T3 adds latency without measurable benefit on this workload. Alex Rivera (trainee, minimal) still hit 59.5% fill purely from playbook inheritance — proof the memory carries knowledge when the model is capable.</p>
|
||||
|
||||
<div class="ref"><strong>Code:</strong> crates/vectord/src/{hnsw.rs, autotune.rs, agent.rs, promotion.rs} · tests/multi-agent/{agent.ts, scenario.ts} · config/models.json · ADR-019</div>
|
||||
</div>
|
||||
|
||||
<!-- ═══ 4. CONTRACT INFERENCE ═══ -->
|
||||
<div class="chapter" id="ch4">
|
||||
<div class="num">Chapter 4</div>
|
||||
<h2>Contract inference from external signal</h2>
|
||||
<div class="lede">Most CRMs wait for a contract to land. This system watches upstream demand and pre-builds the ranking before the contract lands.</div>
|
||||
|
||||
<p>The concrete example running on <code>devop.live/lakehouse</code> is Chicago Department of Buildings permit data (public Socrata API). Every permit is a signal that construction — and therefore staffing — is coming.</p>
|
||||
|
||||
<h3>Flow</h3>
|
||||
<div class="step"><div class="n">1</div><div class="body"><strong>Fetch.</strong> <code>/intelligence/market</code> and <code>/intelligence/permit_contracts</code> hit <code>data.cityofchicago.org/resource/ydr8-5enu.json</code> live. No caching of permit data — every page load is fresh.</div></div>
|
||||
<div class="step"><div class="n">2</div><div class="body"><strong>Map work_type → role.</strong> Industry dictionary: "Electrical Work" → "Electrician", "Masonry Work" → "Production Worker", "Mechanical Work" → "Maintenance Tech", etc.</div></div>
|
||||
<div class="step"><div class="n">3</div><div class="body"><strong>Derive worker count.</strong> Heuristic: ~1 worker per $150K of permit cost, capped 2-8 per contract for staffing realism. Operator-configurable when real client history is available.</div></div>
|
||||
<div class="step"><div class="n">4</div><div class="body"><strong>Derive timeline.</strong> Permit issued → construction starts ~45 days later → staffing window opens ~14 days before construction. Classifies each permit as <code>overdue</code>, <code>urgent</code>, <code>soon</code>, <code>scheduled</code>.</div></div>
|
||||
<div class="step"><div class="n">5</div><div class="body"><strong>Run hybrid search against the bench.</strong> For each derived contract, <code>POST /vectors/hybrid</code> with <code>sql_filter</code> on role+state+city+availability, <code>use_playbook_memory: true</code>, <code>playbook_memory_k: 200</code>. Returns top-5 candidates with boost + citations.</div></div>
|
||||
<div class="step"><div class="n">6</div><div class="body"><strong>Query the meta-index.</strong> <code>POST /vectors/playbook_memory/patterns</code> aggregates traits across similar past playbooks — recurring certs, skills, archetype, reliability distribution. Surfaces signal the operator didn't query for.</div></div>
|
||||
<div class="step"><div class="n">7</div><div class="body"><strong>Render on the dashboard.</strong> Each card shows permit + derived contract + top 3 candidates with memory chips + discovered pattern + urgency. All of this pre-computed before any staffer opens the UI.</div></div>
|
||||
|
||||
<h3>Coverage forecast</h3>
|
||||
<p><code>/intelligence/staffing_forecast</code> aggregates the last 30 days of permits into predicted role-level demand, joins against the IL bench supply, computes coverage %, and classifies each role as <code>critical</code> / <code>tight</code> / <code>watch</code> / <code>ok</code>. The dashboard's top panel renders this — staffers see supply gaps before they query.</p>
|
||||
</div>
|
||||
|
||||
<!-- ═══ 5. WHAT A CRM CAN'T DO ═══ -->
|
||||
<div class="chapter" id="ch5">
|
||||
<div class="num">Chapter 5</div>
|
||||
<h2>What a CRM can't do (and why)</h2>
|
||||
<div class="lede">A CRM stores. This system infers, predicts, re-ranks, and compounds. The six capabilities below are load-bearing — missing any of them is the gap between "software that logs calls" and "software that makes the next call better."</div>
|
||||
|
||||
<div class="card accent-b">
|
||||
<table class="plain">
|
||||
<thead><tr><th>Capability</th><th>CRM</th><th>This system</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Store candidate records</td><td>Yes</td><td>Yes (workers_500k, candidates)</td></tr>
|
||||
<tr><td>Search by structured field</td><td>Yes</td><td>Yes (DataFusion SQL, sub-100ms on 3M rows)</td></tr>
|
||||
<tr><td>Search by semantic meaning</td><td>No</td><td>Yes (HNSW + nomic-embed-text)</td></tr>
|
||||
<tr><td>Combine SQL filter + semantic rank</td><td>No</td><td>Yes (<code>/vectors/hybrid</code>)</td></tr>
|
||||
<tr><td>Boost workers based on past success</td><td>No</td><td>Yes (Phase 19 playbook_memory)</td></tr>
|
||||
<tr><td>Penalize workers based on past failure</td><td>No</td><td>Yes (<code>/log_failure</code> + <code>0.5<sup>n</sup></code> penalty)</td></tr>
|
||||
<tr><td>Surface traits across past fills</td><td>No</td><td>Yes (<code>/vectors/playbook_memory/patterns</code>)</td></tr>
|
||||
<tr><td>Predict staffing demand from external data</td><td>No</td><td>Yes (Chicago permit feed + 30-day rolling forecast)</td></tr>
|
||||
<tr><td>Count down to staffing deadline per contract</td><td>No</td><td>Yes (permit issue_date + heuristic timeline)</td></tr>
|
||||
<tr><td>Explain why each candidate ranked</td><td>No</td><td>Yes (boost chip + narrative citations + memory pattern)</td></tr>
|
||||
<tr><td>Improve ranking from operator actions</td><td>No</td><td>Yes (every Call/SMS/No-show click → re-rank signal)</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- ═══ 6. HOW IT GETS BETTER ═══ -->
|
||||
<div class="chapter" id="ch6">
|
||||
<div class="num">Chapter 6</div>
|
||||
<h2>How it gets better over time</h2>
|
||||
<div class="lede">Compounding learning across seven paths. The first three are automatic background loops. Paths 4-7 landed 2026-04-21 and turn the system into a reinforcement-learning pipeline: outcomes → knowledge base → pathway recommendations → cloud rescue → competence-weighted retrieval → observer analysis. All seven happen without operator intervention.</div>
|
||||
|
||||
<h3>Path 1 — Playbook boost with geo + role prefilter (Phase 19 + refinement)</h3>
|
||||
<p>Every sealed fill is seeded to <code>playbook_memory</code>. The boost fires inside <code>/vectors/hybrid</code> when <code>use_playbook_memory: true</code>. Math, tightened 2026-04-21 after a diagnostic pass found globally-ranked playbooks were missing the SQL-filtered candidate pool entirely:</p>
|
||||
<pre>per_worker = cosine(query_emb, playbook_emb) × 0.5 × e^(-age/30) × 0.5^failures / n_workers
|
||||
boost[(city, state, name)] = min(Σ per_worker, 0.25)</pre>
|
||||
<p><strong>Multi-strategy retrieval (new):</strong> before cosine, <code>compute_boost_for_filtered_with_role(target_geo, target_role)</code> prefilters to same-city playbooks, then gives exact (role, city, state) matches similarity=1.0 and fills up to half the top-k. Cosine fills the rest. Mirrors 2026 Mem0/Zep guidance on parallel-strategy rerank.</p>
|
||||
<p><strong>Measured lift:</strong> before geo-filter, Nashville Welder query returned <code>boosts=170 matched=0</code> (zero intersection with candidate pool). After: <code>boosts=36 matched=11</code>. On the Riverfront Steel scenario, total playbook citations went from 2 → 28 per run — a 14× delta on identical inputs. The diagnostic log <code>playbook_boost: boosts=N sources=N parsed=N matched=N target_geo=? target_role=?</code> runs on every hybrid call so the class of silent-miss bug stays visible.</p>
|
||||
|
||||
<h3>Path 2 — Pattern discovery (meta-index)</h3>
|
||||
<p><code>/vectors/playbook_memory/patterns</code> goes beyond "who was endorsed" to answer "what did past similar fills have in common?" Aggregates recurring certifications, skills, archetype, reliability distribution across the top-K semantically similar playbooks. Surfaces signal the operator didn't explicitly query for.</p>
|
||||
|
||||
<h3>Path 3 — Autotune agent</h3>
|
||||
<p>The <code>vectord::agent</code> background task runs continuously. Watches the HNSW trial journal, proposes configs, executes trials, promotes Pareto winners — without human intervention. Operator sees "the index got faster overnight" and doesn't know why. The journal knows why.</p>
|
||||
|
||||
<h3>Path 4 — Knowledge Base + pathway recommender (Phase 22)</h3>
|
||||
<p>Meta-layer over playbook_memory. Files under <code>data/_kb/</code>:</p>
|
||||
<ul>
|
||||
<li><code>signatures.jsonl</code> — sig_hash + embedding of every run's event shape</li>
|
||||
<li><code>outcomes.jsonl</code> — per-run summary (models, fill rate, turns, citations, rescue stats, staffer, elapsed)</li>
|
||||
<li><code>pathway_recommendations.jsonl</code> — AI-synthesized advice for the next run of a similar sig</li>
|
||||
<li><code>error_corrections.jsonl</code> — fail→succeed deltas on the same sig</li>
|
||||
<li><code>config_snapshots.jsonl</code> — hash of active models + tool_level at each run</li>
|
||||
</ul>
|
||||
<p>Cycle: scenario ends → <code>kb.indexRun()</code> appends outcome → <code>kb.recommendFor(nextSpec)</code> finds k-NN signatures, feeds outcome history to an overview model, writes structured JSON advice → next scenario reads it via <code>kb.loadRecommendation(spec)</code> and injects <code>pathway_notes</code> into the executor's context alongside prior T3 lessons.</p>
|
||||
|
||||
<h3>Path 5 — Cloud rescue on failure (Phase 22 item B)</h3>
|
||||
<p>When an event fails (drift abort, JSON parse, pool exhaustion) and cloud T3 is enabled, <code>requestCloudRemediation()</code> feeds the full failure trace — SQL filters attempted, row counts, reviewer drift notes, gap signals, contract terms — to <code>gpt-oss:120b</code> on Ollama Cloud. Cloud returns structured <code>{retry, new_city, new_state, new_role, new_count, rationale}</code>. Event retries once with the pivot. Verified on stress_01: Gary IN (zero workers indexed) misplacement → cloud proposed South Bend IN → retry filled 1/1.</p>
|
||||
|
||||
<h3>Path 6 — Staffer competence-weighted retrieval (Phase 23)</h3>
|
||||
<p>Answers "who handled this" as a first-class matrix-index dimension. Each scenario carries <code>staffer: {id, name, tenure_months, role, tool_level}</code>. After every run, <code>recomputeStafferStats(staffer_id)</code> aggregates their fill_rate, turn efficiency, citation density, rescue rate into a single <code>competence_score</code> (0.45·fill + 0.20·turn_eff + 0.20·cites + 0.15·rescue).</p>
|
||||
<p><code>findNeighbors</code> returns <code>weighted_score = cosine × max_staffer_competence</code> — top-performer playbooks rank above juniors' on similar scenarios. Auto-discovery emerges: running 4 staffers × 3 contracts × 3 rounds surfaced Rachel D. Lewis (Welder Nashville) with 18 endorsements across all 4 staffers, Angela U. Ward (Machine Op Indianapolis) with 19 — reliable-performer labels the system built without human tagging.</p>
|
||||
|
||||
<h3>Path 7 — Observer outcome ingest (Phase 24)</h3>
|
||||
<p>Observer runs as <code>lakehouse-observer.service</code>, now with an HTTP listener on <code>:3800</code>. Scenarios POST per-event outcomes to <code>/event</code> with full provenance (staffer_id, sig_hash, event_kind, role, city, state, rescue flags). Observer's ERROR_ANALYZER and PLAYBOOK_BUILDER loops consume them alongside MCP-wrapped ops. Persistence switched from the old <code>/ingest/file</code> REPLACE path to an append-only <code>data/_observer/ops.jsonl</code> journal so the trace survives across restarts.</p>
|
||||
|
||||
<h3>Input normalizer + unified memory query</h3>
|
||||
<p>Two surfaces added 2026-04-21 to make the memory stack respond coherently to any input shape:</p>
|
||||
<ul>
|
||||
<li><code>normalizeInput(raw)</code> — accepts structured JSON, natural language, or mixed. Three-tier: structured fast-path → regex (handles "need 3 welders in Nashville, TN" in 0ms without an LLM call) → qwen3 LLM fallback for low-signal inputs.</li>
|
||||
<li><code>POST /memory/query</code> — one endpoint, returns every memory surface in a single bundle: <code>playbook_workers</code> (with boost + citations), <code>pathway_recommendation</code> (KB), <code>neighbor_signatures</code> (competence-weighted), <code>prior_lessons</code> (T3 overseer history), <code>top_staffers</code> (leaderboard), <code>discovered_patterns</code> (auto-surfaced reliable workers for this role+city), <code>latency_ms</code> (per-source). Natural-language query end-to-end: 319ms.</li>
|
||||
</ul>
|
||||
|
||||
<h3>Honest gaps — what we can still implement</h3>
|
||||
<p>Three of the five 2026-era memory findings remain unwired. Flagged for near-term implementation, not hidden:</p>
|
||||
<ul>
|
||||
<li><strong>Zep-style validity windows</strong> — playbook entries have timestamps but no <code>valid_until</code> or <code>schema_fingerprint</code>. Load-bearing: when a schema migration changes a column, stale playbooks silently keep boosting. Biggest-value remaining fix.</li>
|
||||
<li><strong>Mem0-style UPDATE / DELETE / NOOP ops</strong> — <code>/seed</code> only ADDs. Same (operation, date) pair appends a duplicate instead of refining an existing entry. Playbook file grows faster than necessary.</li>
|
||||
<li><strong>Letta working-memory hot cache</strong> — every query scans all 1560 playbook entries from disk. Cheap today at 1.5K, not at 100K. Solution: LRU in-process cache for the last N playbooks or the current sig's neighborhood.</li>
|
||||
</ul>
|
||||
<p>Validity windows is next — preserves the trust signal (boost only fires on playbooks that are still true given the current schema) rather than the latency signal (which the current scale doesn't need yet).</p>
|
||||
|
||||
<div class="ref"><strong>Code:</strong> crates/vectord/src/{playbook_memory.rs, service.rs} · tests/multi-agent/{kb.ts, memory_query.ts, normalize.ts, scenario.ts} · mcp-server/{observer.ts, index.ts} · data/_kb/ · data/_observer/</div>
|
||||
</div>
|
||||
|
||||
<!-- ═══ 7. SCALE STORY ═══ -->
|
||||
<div class="chapter" id="ch7">
|
||||
<div class="num">Chapter 7</div>
|
||||
<h2>Scale story — 20 staffers, 300 contracts, a surge</h2>
|
||||
<div class="lede">What happens when the demo-level load becomes the production-level load, and midday a client pushes 20 more contracts plus a 1M-row ATS delta. Honest: some of this is architectural headroom, not measured scale. The designed behaviors are below.</div>
|
||||
|
||||
<h3>20 concurrent staffers</h3>
|
||||
<p><strong>Axum is async.</strong> The gateway handles concurrent requests on Tokio with work-stealing. No per-request thread. Tested at 10 parallel queries in 82ms total on this hardware.</p>
|
||||
<p><strong>Per-staffer profile isolation.</strong> Each staffer activates their own profile (Phase 17) or workspace (Phase 8.5). Profile scopes their search to bound datasets. Workspace carries their in-progress contracts across sessions.</p>
|
||||
<p><strong>Per-client blacklists.</strong> Auto-applied when the caller passes <code>client: "X"</code> on <code>/search</code>. Staffer A filling for Acme never sees Acme's flagged workers. Staffer B filling for MidState sees them normally.</p>
|
||||
|
||||
<h3>300 active contracts</h3>
|
||||
<p><strong>SQL on <code>job_orders</code> is cheap.</strong> 300 rows is nothing — a scan is microseconds.</p>
|
||||
<p><strong>Workspace per contract.</strong> Each contract gets its own workspace with saved searches, shortlists, activity log. Zero-copy handoff between staffers (pointer swap, not data copy).</p>
|
||||
<p><strong>Forecast remains coherent.</strong> <code>/intelligence/staffing_forecast</code> aggregates 30-day permit data regardless of contract count. The bench supply query (<code>GROUP BY role</code> over workers_500k) is a single sub-second SQL.</p>
|
||||
|
||||
<h3>Midday surge: +20 contracts, +1M profiles</h3>
|
||||
<p>The delta arrives at 12:30. Here's what happens in the following minutes:</p>
|
||||
<div class="step"><div class="n">1</div><div class="body"><strong>+20 contracts via /ingest/db or /ingest/file.</strong> Parsed, schema-checked, Parquet-written, catalog-registered. No queries blocked — register holds a write lock across the manifest write only.</div></div>
|
||||
<div class="step"><div class="n">2</div><div class="body"><strong>+1M worker profiles arrives as delta to workers_500k.</strong> Append-log pattern (ADR-018) means the new rows write to a fresh batch file — base Parquet is NOT rewritten. Queries against workers_500k immediately merge-on-read the new batches.</div></div>
|
||||
<div class="step"><div class="n">3</div><div class="body"><strong>Embeddings marked stale.</strong> The vector index for workers_500k_v1 now has 1M rows it hasn't seen. <code>mark_embeddings_stale</code> flips the flag.</div></div>
|
||||
<div class="step"><div class="n">4</div><div class="body"><strong>Incremental refresh fires.</strong> <code>POST /vectors/refresh/workers_500k</code> reads only the new rows (diff against existing embeddings), embeds them in batches of 64 via Ollama, writes delta embedding Parquet. Measured on threat_intel: 34 new rows in 970ms (6× faster than full re-embed).</div></div>
|
||||
<div class="step"><div class="n">5</div><div class="body"><strong>Search degrades gracefully.</strong> During the refresh, searches against workers_500k_v1 still work — they serve from the old embeddings. Brute-force cosine over new-rows-without-embeddings is allowed but costs more. HNSW rebuild happens after all embeds complete.</div></div>
|
||||
<div class="step"><div class="n">6</div><div class="body"><strong>Hot-swap promotion.</strong> When the new index is ready, <code>promotion_registry</code> atomically flips the active pointer. Next search hits the new config. Rollback stays available.</div></div>
|
||||
<div class="step"><div class="n">7</div><div class="body"><strong>Autotune re-enters the loop.</strong> The agent queue picks up a <code>DatasetAppended</code> trigger and schedules a fresh HNSW trial cycle against the expanded index.</div></div>
|
||||
|
||||
<h3>Known pain points at this scale</h3>
|
||||
<ul>
|
||||
<li><strong>Ollama inference is serial.</strong> Embedding 1M rows at ~50 chunks/sec through nomic-embed-text = ~6 hours. Acceptable for overnight refresh, not for "immediate." Mitigated by incremental refresh (only deltas).</li>
|
||||
<li><strong>RAM ceiling on HNSW.</strong> Around 5M vectors × 768d, HNSW stops fitting in 128GB comfortably. Mitigation: per-profile <code>vector_backend: lance</code> flip — disk-resident IVF_PQ scales past the RAM line (ADR-019).</li>
|
||||
<li><strong>VRAM ceiling for model variety.</strong> A4000 16GB holds 1-2 loaded models. Multi-model recruiter surfaces are a sequential swap, not parallel (Ollama <code>keep_alive=0</code>). Phase 17 profile activation unloads the prior model on swap.</li>
|
||||
<li><strong>playbook_memory growth.</strong> 1936 entries today. Phase 25 (2026-04-21) added retirement via <code>valid_until</code> + <code>schema_fingerprint</code> fields + <code>POST /vectors/playbook_memory/retire</code> endpoint (manual or schema-drift triggered). Active vs retired split surfaced on <code>GET /vectors/playbook_memory/status</code>. Brute-force cosine still sub-ms at current size; Letta-style working-memory hot cache deferred until entry count crosses ~100K.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
<!-- ═══ 8. ERROR SURFACES ═══ -->
|
||||
<div class="chapter" id="ch8">
|
||||
<div class="num">Chapter 8</div>
|
||||
<h2>Error surfaces & recovery</h2>
|
||||
<div class="lede">Every failure mode has a named surface, a structured response, and a recovery path. No silent failures.</div>
|
||||
|
||||
<table class="plain">
|
||||
<thead><tr><th>Failure mode</th><th>Surface / response</th><th>Recovery</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Ingest receives file with schema mismatch vs existing dataset</td><td><code>409 Conflict</code> with both fingerprints named (ADR-020)</td><td>Re-ingest under a new name, or migrate the existing via Phase 14 schema evolution</td></tr>
|
||||
<tr><td>Bucket unreachable on write</td><td>Hard 503, error journaled to <code>primary://_errors/bucket_errors/</code></td><td><code>GET /storage/errors</code> lists failures; <code>GET /storage/bucket-health</code> shows per-bucket status</td></tr>
|
||||
<tr><td>Bucket unreachable on read</td><td>Rescue bucket fallback, <code>X-Lakehouse-Rescue-Used: true</code> header on response</td><td>Response still succeeds; operator sees rescue flag</td></tr>
|
||||
<tr><td>/log receives name that doesn't exist in workers_500k</td><td>Seed is SKIPPED; response includes <code>rejected_ghost_names: [...]</code> and a note</td><td>Operator sees exactly which names were rejected and why</td></tr>
|
||||
<tr><td>Dual-agent executor malforms tool call</td><td>Result appended to log with <code>error</code> field; counter increments</td><td>After 3 consecutive: abort with full log dump at <code>tests/multi-agent/playbooks/<id>-FAILED.json</code></td></tr>
|
||||
<tr><td>Dual-agent drifts from target</td><td>Reviewer verdict = <code>drift</code>, counter increments</td><td>After 3 consecutive drifts: abort with full log</td></tr>
|
||||
<tr><td>Hybrid search finds zero candidates</td><td>Returns empty <code>sources[]</code> + <code>sql_matches: 0</code></td><td>Gap signal captured by scenario runner; operator prompted to broaden filter</td></tr>
|
||||
<tr><td>Ollama sidecar down</td><td>502 Bad Gateway from aibridge; <code>embed</code> calls fail fast</td><td>Restart: <code>systemctl restart lakehouse-sidecar</code>; vector search falls back to pre-computed embeddings</td></tr>
|
||||
<tr><td>Gateway restart mid-operation</td><td>In-memory state (playbook_memory, HNSW) reloaded from persisted <code>state.json</code> / trial journals</td><td>Zero data loss; catalog, storage, journals are all source-of-truth</td></tr>
|
||||
<tr><td>Schema fingerprint diverges across manifests</td><td><code>catalog::dedupe</code> reports <code>DedupeReport</code> with winner selection (non-null row_count first, then newest updated_at)</td><td><code>POST /catalog/dedupe</code> collapses duplicates idempotently</td></tr>
|
||||
<tr><td>Scenario event fails on zero-supply city</td><td>Cloud rescue (Phase 22 item B) fires — <code>gpt-oss:120b</code> sees SQL filters attempted, row counts, reviewer drift notes, contract terms; returns structured <code>{retry, new_city, new_state, new_role, new_count, rationale}</code></td><td>Retry with pivot runs same executor loop with new geography; verified Gary IN → South Bend IN filled 1/1 after original drift-abort</td></tr>
|
||||
<tr><td>LLM response truncated mid-JSON (thinking model ate token budget)</td><td>Phase 21 <code>generateContinuable()</code> detects via brace-balance + JSON.parse; no silent truncation</td><td>Auto-continue with partial as scratchpad, or geometric backoff if initial call returned empty. Bounded by <code>max_continuations</code>.</td></tr>
|
||||
<tr><td>Schema migration invalidates existing playbooks</td><td>Phase 25 — <code>POST /vectors/playbook_memory/retire</code> with current fingerprint retires all mismatched entries in scope; diagnostic log shows counts</td><td>Retired entries stay in journal for forensics but are skipped by all boost calculations. Scoped by (city, state) so unrelated geos aren't touched.</td></tr>
|
||||
<tr><td>Observer fails to reach scenario outcome stream</td><td>Scenario <code>postObserverEvent()</code> uses 2s <code>AbortSignal.timeout</code>; silent skip if :3800 is down</td><td>Scenario log is still the source of truth; observer re-ingest on next run restores the stream. <code>data/_observer/ops.jsonl</code> is append-only so prior events survive.</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<!-- ═══ 9. PER-STAFFER CONTEXT ═══ -->
|
||||
<div class="chapter" id="ch9">
|
||||
<div class="num">Chapter 9</div>
|
||||
<h2>Per-staffer context</h2>
|
||||
<div class="lede">Twenty staffers don't see the same UI state. Each one's session is shaped by their active profile, their workspaces, their assigned contracts, and their client's blacklists.</div>
|
||||
|
||||
<h3>Active profile (Phase 17)</h3>
|
||||
<p>Scopes every search. A <code>staffing-recruiter</code> profile bound to <code>workers_500k</code> sees only that dataset. A <code>security-analyst</code> profile bound to <code>threat_intel</code> cannot see worker data. <code>GET /vectors/profile/<id>/audit</code> records every tool invocation by model identity.</p>
|
||||
|
||||
<h3>Workspace (Phase 8.5)</h3>
|
||||
<p>Per-contract state. Each workspace has daily/weekly/monthly tiers, saved searches, shortlists, activity logs. Survives across sessions. Instant zero-copy handoff between staffers — pointer swap, not data copy. Persisted to object storage, rebuilt on startup.</p>
|
||||
|
||||
<h3>Client blacklist</h3>
|
||||
<p>Per-client worker exclusion. Populated via <code>POST /clients/:client/blacklist</code>. Auto-applied when the caller passes <code>client: "X"</code> on <code>/search</code>. JSON-backed; would move to catalog table under real client load.</p>
|
||||
|
||||
<h3>Audit trail</h3>
|
||||
<p>Phase 12 tool registry logs every governed-action invocation (who called what, with what args, when, outcome). <code>GET /tools/audit</code> queryable. Phase 13 access control layers on top — role-based field masking, query audit log.</p>
|
||||
|
||||
<h3>Daily summary per staffer</h3>
|
||||
<p>Workspace activity log + per-staffer filter on the event journal gives <strong>"what did Sarah do today"</strong> as a direct query. The foundation for shift-handoff reports.</p>
|
||||
|
||||
<h3>Staffer identity + competence-weighted retrieval (Phase 23)</h3>
|
||||
<p>Each scenario run carries an explicit <code>staffer: {id, name, tenure_months, role, tool_level}</code>. The KB aggregates per-staffer stats (<code>data/_kb/staffers.jsonl</code>) that roll up into a single <code>competence_score</code>:</p>
|
||||
<pre>competence_score = 0.45·fill_rate + 0.20·turn_efficiency + 0.20·citation_density + 0.15·rescue_rate</pre>
|
||||
<p>When any query runs <code>kb.findNeighbors(spec, k)</code>, the ranking isn't just cosine similarity — it's <code>weighted_score = cosine × max_staffer_competence</code> over the best coordinator who ran that signature. Senior staffers' playbooks surface above juniors' on similar scenarios, even when the juniors' scenario was marginally closer in embedding space.</p>
|
||||
<p>The <code>tool_level</code> knob (full / local / basic / minimal) controls which tiers are available to a given staffer's runs. See Ch 3 for the mapping. Variance is real and measurable: 36-run demo produced a 46pt fill-rate delta between James (local tools, 93%) and Alex (minimal tools, 60%) on identical contracts.</p>
|
||||
|
||||
<h3>Auto-discovered reliable-performer labels</h3>
|
||||
<p>A second-order output of the competence path: when multiple staffers independently endorse the same worker on similar-signature playbooks, that worker accumulates cross-staffer endorsements. <code>scripts/kb_staffer_report.py</code> surfaces them — after 36 runs, Rachel D. Lewis (Welder Nashville) had 18 endorsements across 4 staffers, Angela U. Ward (Machine Op Indianapolis) 19. These are high-confidence "reliable" labels the system produced without human tagging. The UI could badge these workers on future queries; today they're visible via <code>/memory/query</code>'s <code>discovered_patterns</code> bundle.</p>
|
||||
</div>
|
||||
|
||||
<!-- ═══ 10. A DAY IN THE LIFE ═══ -->
|
||||
<div class="chapter" id="ch10">
|
||||
<div class="num">Chapter 10</div>
|
||||
<h2>A day in the life — from morning brief to EOD retrospective</h2>
|
||||
<div class="lede">Concrete operator timeline. Every step touches a real endpoint that exists today.</div>
|
||||
|
||||
<div class="step"><div class="n">07:00</div><div class="body"><strong>Overnight housekeeping.</strong> Scheduled ingest runs — the configured cron picks up the client's latest ATS CSV delta, runs it through the pipeline in Ch2, marks workers_500k embeddings stale. Autotune agent promotes any Pareto-winner HNSW configs from overnight trials.</div></div>
|
||||
|
||||
<div class="step"><div class="n">07:30</div><div class="body"><strong>Embedding refresh.</strong> Background job re-embeds the new rows. Old index keeps serving. Hot-swap promotes when done.</div></div>
|
||||
|
||||
<div class="step"><div class="n">08:00</div><div class="body"><strong>Sarah (staffer) opens devop.live/lakehouse.</strong> Page loads in ~3s. Forecast panel shows: "$275M construction coming, 4 tight roles this week." Live Contracts section shows 6 Chicago permits with proposed fills + boost chips + pattern signals.</div></div>
|
||||
|
||||
<div class="step"><div class="n">08:15</div><div class="body"><strong>Sarah drills into a $5M permit.</strong> Top candidate card: Carmen Green, Endorsed · 3 playbooks chip, boost +0.166, pattern line reads "leader archetype · 47% OSHA-10." Sarah hovers the chip — narrative tooltip: "filled Welder x2 in Toledo (2026-04-15), Welder x1 in Toledo (2026-04-18)."</div></div>
|
||||
|
||||
<div class="step"><div class="n">08:30</div><div class="body"><strong>Sarah calls Carmen.</strong> Clicks Call button → <code>/log</code> fires → <code>playbook_memory.seed</code> → <code>persist_sql</code> → successful_playbooks_live grows by one. Button flashes "Logged" for 1.4s. No modal, no form, no second click.</div></div>
|
||||
|
||||
<div class="step"><div class="n">09:00</div><div class="body"><strong>Kim (another staffer) opens the same UI.</strong> Her profile loads. Her workspaces show her own contracts. She searches "reliable forklift Chicago" — MEMORY chip shows the pattern discovered across Sarah's morning work AND prior fills. Carmen, already logged by Sarah, shows up with an updated citation count.</div></div>
|
||||
|
||||
<div class="step"><div class="n">12:30</div><div class="body"><strong>Client pushes 20 new contracts + 1M ATS delta.</strong> Ch7 scale flow fires. Ingest in seconds; embedding refresh kicks off as a background job. Searches continue against old embeddings.</div></div>
|
||||
|
||||
<div class="step"><div class="n">14:00</div><div class="body"><strong>Emergency: worker Dave no-showed.</strong> Sarah clicks No-show button on Dave's card → <code>/log_failure</code> → <code>mark_failed</code> records a penalty. Next similar query dampens Dave's boost by 0.5. Sarah continues the refill — the refill excludes Dave and the 2 others already booked for this shift.</div></div>
|
||||
|
||||
<div class="step"><div class="n">15:00</div><div class="body"><strong>New embeddings live.</strong> Hot-swap promotion. Searches now see all 1M new profiles. Sarah's noon query re-run would produce different top-5.</div></div>
|
||||
|
||||
<div class="step"><div class="n">17:00</div><div class="body"><strong>End-of-day retrospective.</strong> Any staffer who ran <code>tests/multi-agent/scenario.ts</code> gets <code>report.md</code> auto-generated. Workspace activity logs aggregate per staffer. <code>GET /vectors/playbook_memory/status</code> shows active vs retired counts. KB indexes the run (<code>kb.indexRun</code>) and the overview model synthesizes a pathway recommendation for the next matching signature. Every event outcome has already streamed to <code>lakehouse-observer.service</code> on :3800 for ERROR_ANALYZER + PLAYBOOK_BUILDER consumption.</div></div>
|
||||
|
||||
<div class="step"><div class="n">17:15</div><div class="body"><strong>Kim fires a natural-language query from the search box.</strong> "need 3 forklift operators in Joliet by Monday" → <code>POST /memory/query</code> on the Bun MCP. Regex normalizer extracts role / city / count / deadline / intent in 0ms (no LLM call). The unified response returns playbook workers (auto-surfaced reliable performers for Joliet Forklift with citation counts), pathway recommendation from KB, prior T3 lessons for Joliet, and top staffers by competence — all in ~300ms.</div></div>
|
||||
|
||||
<div class="step"><div class="n">22:00</div><div class="body"><strong>Overnight trial cycle.</strong> Autotune agent continues in the background. Trial journal grows. KB's <code>detectErrorCorrections</code> scans today's outcomes for fail→succeed deltas on the same signature; any correction gets logged to <code>data/_kb/error_corrections.jsonl</code> with the config diff. Tomorrow morning, the system is measurably better at something it got asked about today.</div></div>
|
||||
|
||||
<h3>SMS + email drafts in the pipeline</h3>
|
||||
<p>After each sealed fill (via scenario.ts or manual <code>/log</code> flow with downstream hooks), <code>generateArtifacts</code> in the scenario runner produces: (a) one SMS per worker (TO: Name, message under 180 chars), (b) one client confirmation email. Drafts are saved to <code>sms.md</code> and <code>emails.md</code> under the scenario output dir. Ollama drafts them; the staffer reviews and sends. No auto-send; human-in-the-loop.</p>
|
||||
</div>
|
||||
|
||||
<!-- ═══ 11. LIMITS & NON-GOALS ═══ -->
|
||||
<div class="chapter" id="ch11">
|
||||
<div class="num">Chapter 11</div>
|
||||
<h2>Known limits & non-goals</h2>
|
||||
<div class="lede">Honesty is a feature. Everything below is either deferred or explicitly out of scope.</div>
|
||||
|
||||
<h4>Deferred — real architectural work, just not shipped yet</h4>
|
||||
<ul>
|
||||
<li><strong>Rate / margin awareness.</strong> Worker pay expectations vs contract bill rate not modeled. Requires adding <code>pay_rate</code> to workers, <code>bill_rate</code> to contracts, and a filter + warning path. Partially addressed via <code>ContractTerms.budget_per_hour_max</code> passed to T3/rescue prompts, but the match-time filter isn't wired yet.</li>
|
||||
<li><strong>Mem0-style UPDATE / DELETE / NOOP operations on playbooks.</strong> Today <code>/seed</code> only ADDs. Same <code>(operation, date)</code> pair appends a duplicate instead of refining an existing entry. Phase 26 item — cheap to add, moderate payoff.</li>
|
||||
<li><strong>Letta working-memory hot cache.</strong> Every boost query scans all active playbook entries from in-memory state. 1.9K today; cheap. Will bite somewhere north of 100K. LRU for the last-N playbooks or current-sig neighborhood deferred until that ceiling approaches.</li>
|
||||
<li><strong>Chunking cache (Phase 21 Rust port).</strong> TS primitives <code>generateContinuable</code> + <code>generateTreeSplit</code> are wired, but <code>crates/aibridge/src/{continuation.rs, tree_split.rs}</code> + <code>crates/storaged/src/chunk_cache.rs</code> remain queued. Gateway-side callers currently don't have the same protection against silent truncation that the TS test harness does.</li>
|
||||
<li><strong>Confidence calibration.</strong> Top-K is a rank, not a probability. No calibrated "85% likely to accept" score. Requires outcome-labeled training data.</li>
|
||||
<li><strong>Neural re-ranker.</strong> Phase 19 is statistical + semantic (now with geo + role prefilter, Phase 25 retirement). A (query, candidate, outcome)-trained re-ranker is deferred only if the statistical floor plateaus below usable recall — current 14× citation lift on identical inputs suggests it hasn't.</li>
|
||||
<li><strong>Observer → autotune feedback wire.</strong> Phase 24 streams scenario outcomes into <code>data/_observer/ops.jsonl</code>; autotune agent still runs on its own HNSW-trial schedule and hasn't subscribed to the outcome metric stream yet. Phase 26+ item — connects the last loop.</li>
|
||||
<li><strong>call_log cross-reference.</strong> Infrastructure present; current synthetic candidates table is too small to cross-ref. Fixes when real ATS lands.</li>
|
||||
</ul>
|
||||
|
||||
<h4>Non-goals — explicitly out of scope</h4>
|
||||
<ul>
|
||||
<li><strong>Cloud deployment.</strong> Local-first by design. Works offline after setup.</li>
|
||||
<li><strong>Full ACID transactions.</strong> Single-writer model is sufficient; Delta Lake-grade MVCC is deliberately not attempted.</li>
|
||||
<li><strong>Real-time streaming / CDC.</strong> Batch ingest is the model. Scheduled refresh, not transactional replication.</li>
|
||||
<li><strong>Replacing the CRM.</strong> This is the analytical + AI layer <em>behind</em> the CRM. Operational CRUD stays with the existing system.</li>
|
||||
<li><strong>Custom file formats.</strong> Parquet for datasets, sidecar indexes for vectors. No proprietary formats (ADR-008, ADR-018 reaffirm).</li>
|
||||
<li><strong>Hard multi-tenant isolation.</strong> Profiles and federation provide soft isolation. Adversarial multi-tenant is not a goal — this system assumes a single-trust operator.</li>
|
||||
</ul>
|
||||
|
||||
<div class="narr">
|
||||
<strong>Overall bet.</strong> The substrate is conservative: Parquet + DataFusion + HNSW + Ollama + object storage. Every layer is replaceable, open, auditable. The intelligence layer (playbook_memory, patterns, autotune) is statistical, not neural — cheaper, explainable, rebuildable from the journal alone. If the statistical floor plateaus below what a real client needs, Phase 20+ adds neural re-rank on top. We don't make that call until measurement demands it.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="footer">Lakehouse spec · v2 2026-04-21 · Phases 19-25 shipped (playbook boost, model matrix, continuation, KB, staffer competence, observer ingest, validity windows) · maintained from <code>docs/DECISIONS.md</code> · <a href="proof">architecture live-tested</a> · <a href="console">walkthrough</a></div>
|
||||
|
||||
</body></html>
|
||||
@@ -0,0 +1,52 @@
|
||||
/**
|
||||
* Langfuse tracing for the Lakehouse agent gateway.
|
||||
*
|
||||
* Every LLM call gets traced: model, prompt, output, latency, tokens.
|
||||
* The observer uses this to build a picture of what's working.
|
||||
*
|
||||
* Langfuse UI: http://localhost:3001
|
||||
* Login: j@lakehouse.local / lakehouse2026
|
||||
*/
|
||||
|
||||
import { Langfuse } from "langfuse";
|
||||
|
||||
const langfuse = new Langfuse({
|
||||
publicKey: process.env.LANGFUSE_PUBLIC_KEY || "pk-lf-staffing",
|
||||
secretKey: process.env.LANGFUSE_SECRET_KEY || "sk-lf-staffing-secret",
|
||||
baseUrl: process.env.LANGFUSE_URL || "http://localhost:3001",
|
||||
enabled: true,
|
||||
});
|
||||
|
||||
export type TraceContext = ReturnType<typeof langfuse.trace>;
|
||||
|
||||
export function startTrace(name: string, input?: any, metadata?: any) {
|
||||
return langfuse.trace({ name, input, metadata });
|
||||
}
|
||||
|
||||
export function logGeneration(
|
||||
trace: TraceContext, name: string,
|
||||
opts: { model: string; prompt: string; completion: string;
|
||||
duration_ms: number; tokens_in?: number; tokens_out?: number; },
|
||||
) {
|
||||
trace.generation({
|
||||
name, model: opts.model, input: opts.prompt, output: opts.completion,
|
||||
usage: { promptTokens: opts.tokens_in, completionTokens: opts.tokens_out },
|
||||
metadata: { duration_ms: opts.duration_ms },
|
||||
});
|
||||
}
|
||||
|
||||
export function logSpan(trace: TraceContext, name: string, input: any, output: any, duration_ms: number) {
|
||||
trace.span({ name, input, output, metadata: { duration_ms } });
|
||||
}
|
||||
|
||||
export function logRetrieval(trace: TraceContext, name: string, query: string, results: any[], duration_ms: number) {
|
||||
trace.span({ name, input: { query }, output: { results_count: results.length }, metadata: { duration_ms, type: "retrieval" } });
|
||||
}
|
||||
|
||||
export function scoreTrace(trace: TraceContext, name: string, value: number, comment?: string) {
|
||||
trace.score({ name, value, comment });
|
||||
}
|
||||
|
||||
export async function flush() { await langfuse.flushAsync(); }
|
||||
export async function shutdown() { await langfuse.shutdownAsync(); }
|
||||
export { langfuse };
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
// Environment setup & latest features
|
||||
"lib": ["ESNext"],
|
||||
"target": "ESNext",
|
||||
"module": "Preserve",
|
||||
"moduleDetection": "force",
|
||||
"jsx": "react-jsx",
|
||||
"allowJs": true,
|
||||
|
||||
// Bundler mode
|
||||
"moduleResolution": "bundler",
|
||||
"allowImportingTsExtensions": true,
|
||||
"verbatimModuleSyntax": true,
|
||||
"noEmit": true,
|
||||
|
||||
// Best practices
|
||||
"strict": true,
|
||||
"skipLibCheck": true,
|
||||
"noFallthroughCasesInSwitch": true,
|
||||
"noUncheckedIndexedAccess": true,
|
||||
"noImplicitOverride": true,
|
||||
|
||||
// Some stricter flags (disabled by default)
|
||||
"noUnusedLocals": false,
|
||||
"noUnusedParameters": false,
|
||||
"noPropertyAccessFromIndexSignature": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,487 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en"><head>
|
||||
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>Lakehouse — Workspaces</title>
|
||||
<style>
|
||||
*{margin:0;padding:0;box-sizing:border-box}
|
||||
body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;color:#b0b8c4;font-size:14px;line-height:1.55;-webkit-font-smoothing:antialiased}
|
||||
a{color:#58a6ff;text-decoration:none}
|
||||
a:hover{color:#79c0ff}
|
||||
|
||||
.bar{background:#0d1117;padding:0 24px;height:56px;border-bottom:1px solid #171d27;display:flex;justify-content:space-between;align-items:center;position:sticky;top:0;z-index:10}
|
||||
.bar h1{font-size:14px;font-weight:600;color:#e6edf3;letter-spacing:-0.2px}
|
||||
.bar nav{display:flex;gap:2px;flex-wrap:wrap}
|
||||
.bar nav a{font-size:12px;color:#545d68;padding:6px 14px;border-radius:6px;transition:all 0.15s}
|
||||
.bar nav a:hover{color:#e6edf3;background:#161b22}
|
||||
.bar nav a.active{color:#e6edf3;background:#1c2333}
|
||||
|
||||
.wrap{max-width:1200px;margin:0 auto;padding:24px 20px 60px}
|
||||
|
||||
h2{color:#e6edf3;font-size:22px;font-weight:700;letter-spacing:-0.3px;margin-bottom:4px}
|
||||
.lede{color:#8b949e;font-size:13px;margin-bottom:18px;line-height:1.6;max-width:720px}
|
||||
|
||||
.layout{display:grid;grid-template-columns:320px 1fr;gap:16px}
|
||||
|
||||
.list{display:flex;flex-direction:column;gap:8px;max-height:calc(100vh - 200px);overflow-y:auto;padding-right:4px}
|
||||
.ws-card{background:#0d1117;border:1px solid #171d27;border-radius:10px;padding:12px 14px;cursor:pointer;transition:all 0.15s}
|
||||
.ws-card:hover{border-color:#30363d;background:#161b22}
|
||||
.ws-card.active{border-color:#1f6feb;background:#0d1d38}
|
||||
.ws-card .name{color:#e6edf3;font-weight:600;font-size:13px;margin-bottom:4px;line-height:1.4}
|
||||
.ws-card .meta{color:#8b949e;font-size:11px}
|
||||
.ws-card .tier{display:inline-block;padding:1px 7px;border-radius:8px;font-size:9px;font-weight:600;margin-right:6px;letter-spacing:0.3px}
|
||||
.tier.daily{background:#2d1b00;color:#d29922;border:1px solid #854d0e}
|
||||
.tier.weekly{background:#0d2340;color:#58a6ff;border:1px solid #1f6feb80}
|
||||
.tier.monthly{background:#1e1047;color:#a78bfa;border:1px solid #5b21b680}
|
||||
.tier.pinned{background:#0d2818;color:#3fb950;border:1px solid #2ea04380}
|
||||
|
||||
.detail{background:#0d1117;border:1px solid #171d27;border-radius:12px;padding:18px;min-height:400px}
|
||||
|
||||
.section{margin-bottom:20px}
|
||||
.section-hdr{display:flex;justify-content:space-between;align-items:center;margin-bottom:10px}
|
||||
.section-hdr .title{color:#545d68;font-size:11px;text-transform:uppercase;letter-spacing:1.2px;font-weight:600}
|
||||
.section-hdr .count{color:#8b949e;font-size:11px}
|
||||
|
||||
.entry{background:#161b22;border-radius:6px;padding:10px 12px;margin-bottom:6px;font-size:12px}
|
||||
.entry .title{color:#e6edf3;font-weight:500;font-size:13px}
|
||||
.entry .meta{color:#8b949e;font-size:11px;margin-top:2px}
|
||||
.entry .sub{color:#545d68;font-size:11px;margin-top:2px;font-family:ui-monospace,Menlo,monospace}
|
||||
|
||||
.empty{color:#484f58;font-size:12px;font-style:italic;padding:10px 12px}
|
||||
|
||||
.btn{padding:8px 14px;background:#1f6feb;border:none;border-radius:6px;color:#fff;font-size:12px;font-weight:600;cursor:pointer}
|
||||
.btn:hover{background:#388bfd}
|
||||
.btn:disabled{opacity:0.4;cursor:not-allowed}
|
||||
.btn.ghost{background:transparent;border:1px solid #21262d;color:#c9d1d9}
|
||||
.btn.ghost:hover{background:#161b22}
|
||||
.btn.green{background:#2ea043}
|
||||
.btn.green:hover{background:#3fb950}
|
||||
.btn.sm{padding:4px 10px;font-size:11px}
|
||||
|
||||
.form{display:flex;flex-direction:column;gap:10px}
|
||||
.form label{color:#8b949e;font-size:11px;text-transform:uppercase;letter-spacing:1px;font-weight:600;margin-bottom:-6px}
|
||||
.form input,.form textarea,.form select{padding:8px 12px;background:#161b22;border:1px solid #21262d;border-radius:6px;color:#e6edf3;font-size:13px;outline:none;font-family:inherit}
|
||||
.form input:focus,.form textarea:focus,.form select:focus{border-color:#388bfd}
|
||||
.form textarea{resize:vertical;min-height:60px;font-family:inherit}
|
||||
.form-row{display:flex;gap:10px}
|
||||
.form-row>*{flex:1}
|
||||
|
||||
.modal-backdrop{position:fixed;inset:0;background:rgba(0,0,0,0.7);z-index:100;display:none;align-items:center;justify-content:center;padding:20px}
|
||||
.modal-backdrop.show{display:flex}
|
||||
.modal{background:#0d1117;border:1px solid #30363d;border-radius:12px;padding:22px;max-width:520px;width:100%;max-height:90vh;overflow-y:auto}
|
||||
.modal h3{color:#e6edf3;font-size:17px;font-weight:700;margin-bottom:12px}
|
||||
|
||||
.hdr-block{padding-bottom:14px;border-bottom:1px solid #171d27;margin-bottom:16px}
|
||||
.hdr-block .wname{color:#e6edf3;font-size:18px;font-weight:700;margin-bottom:4px}
|
||||
.hdr-block .wmeta{color:#8b949e;font-size:12px}
|
||||
.hdr-block .desc{color:#c9d1d9;font-size:13px;margin-top:8px;line-height:1.6}
|
||||
|
||||
.prev-owners{display:flex;flex-wrap:wrap;gap:4px;margin-top:6px}
|
||||
.prev-owners .chip{padding:2px 8px;border-radius:10px;font-size:10px;background:#161b22;color:#8b949e;border:1px solid #21262d}
|
||||
|
||||
.actions{display:flex;gap:8px;flex-wrap:wrap;margin-top:12px}
|
||||
|
||||
.flash{padding:10px 14px;border-radius:6px;margin-top:10px;font-size:12px}
|
||||
.flash.ok{background:#0d2818;color:#86efac;border:1px solid #2ea04360}
|
||||
.flash.err{background:#3a1a1a;color:#fca5a5;border:1px solid #f8514960}
|
||||
|
||||
.activity{max-height:240px;overflow-y:auto}
|
||||
.activity .entry{border-left:2px solid #2ea043}
|
||||
|
||||
.footer{border-top:1px solid #171d27;padding:20px;text-align:center;color:#3d444d;font-size:11px}
|
||||
|
||||
@media(max-width:900px){
|
||||
.layout{grid-template-columns:1fr}
|
||||
.list{max-height:none}
|
||||
.bar nav{font-size:10px}
|
||||
}
|
||||
</style></head>
|
||||
<body>
|
||||
|
||||
<div class="bar">
|
||||
<h1>Lakehouse — Workspaces</h1>
|
||||
<nav>
|
||||
<a href=".">Dashboard</a>
|
||||
<a href="console">Walkthrough</a>
|
||||
<a href="proof">Architecture</a>
|
||||
<a href="spec">Spec</a>
|
||||
<a href="onboard">Onboard</a>
|
||||
<a href="alerts">Alerts</a>
|
||||
<a href="workspaces" class="active">Workspaces</a>
|
||||
</nav>
|
||||
</div>
|
||||
|
||||
<div class="wrap">
|
||||
|
||||
<div style="margin-bottom:18px">
|
||||
<h2>Per-contract workspaces</h2>
|
||||
<div class="lede">
|
||||
Each contract gets its own workspace: saved searches, shortlisted candidates, activity log, tier (daily / weekly / monthly / pinned). Persisted across sessions. One-click zero-copy handoff between staffers — Sarah hands off to Kim, Kim opens the workspace and sees everything Sarah saw. This is the layer that keeps institutional memory per-contract, not per-person.
|
||||
</div>
|
||||
<button class="btn" id="new-ws-btn">New workspace</button>
|
||||
</div>
|
||||
|
||||
<div class="layout">
|
||||
<div class="list" id="ws-list"><div class="empty">Loading workspaces…</div></div>
|
||||
<div class="detail" id="ws-detail">
|
||||
<div class="empty">Select a workspace from the list, or create one.</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<!-- Modals -->
|
||||
<div class="modal-backdrop" id="modal-new">
|
||||
<div class="modal">
|
||||
<h3>New workspace</h3>
|
||||
<form class="form" id="form-new" onsubmit="return false">
|
||||
<label>Contract / fill name</label>
|
||||
<input type="text" id="nw-name" placeholder="e.g. Riverfront Steel — Week 17 fills" required>
|
||||
<label>Description</label>
|
||||
<textarea id="nw-desc" placeholder="What's this contract about?"></textarea>
|
||||
<div class="form-row">
|
||||
<div>
|
||||
<label>Tier</label>
|
||||
<select id="nw-tier">
|
||||
<option value="daily">Daily</option>
|
||||
<option value="weekly" selected>Weekly</option>
|
||||
<option value="monthly">Monthly</option>
|
||||
<option value="pinned">Pinned</option>
|
||||
</select>
|
||||
</div>
|
||||
<div>
|
||||
<label>Owner (staffer)</label>
|
||||
<input type="text" id="nw-owner" placeholder="sarah" value="sarah">
|
||||
</div>
|
||||
</div>
|
||||
<div class="actions">
|
||||
<button type="button" class="btn green" id="nw-create">Create</button>
|
||||
<button type="button" class="btn ghost" data-close="modal-new">Cancel</button>
|
||||
</div>
|
||||
<div id="nw-result"></div>
|
||||
</form>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="modal-backdrop" id="modal-handoff">
|
||||
<div class="modal">
|
||||
<h3>Hand off to another staffer</h3>
|
||||
<div style="color:#8b949e;font-size:12px;margin-bottom:14px;line-height:1.6">
|
||||
Transfers ownership of this workspace. The prior owner is preserved in <code>previous_owners</code> for the audit trail.
|
||||
The recipient sees all saved searches, shortlist entries, and activity from the prior owner — no copy, no migration.
|
||||
</div>
|
||||
<form class="form" onsubmit="return false">
|
||||
<label>New owner</label>
|
||||
<input type="text" id="ho-owner" placeholder="kim">
|
||||
<label>Reason</label>
|
||||
<input type="text" id="ho-reason" placeholder="e.g. end of shift, taking over for Sarah">
|
||||
<div class="actions">
|
||||
<button type="button" class="btn green" id="ho-submit">Hand off</button>
|
||||
<button type="button" class="btn ghost" data-close="modal-handoff">Cancel</button>
|
||||
</div>
|
||||
<div id="ho-result"></div>
|
||||
</form>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="modal-backdrop" id="modal-shortlist">
|
||||
<div class="modal">
|
||||
<h3>Shortlist a candidate</h3>
|
||||
<form class="form" onsubmit="return false">
|
||||
<label>Dataset</label>
|
||||
<input type="text" id="sl-dataset" placeholder="workers_500k" value="workers_500k">
|
||||
<label>Worker ID</label>
|
||||
<input type="text" id="sl-record" placeholder="e.g. 4661 or W500K-4661">
|
||||
<label>Notes</label>
|
||||
<textarea id="sl-notes" placeholder="Why this candidate — skills, past fills, availability window"></textarea>
|
||||
<div class="actions">
|
||||
<button type="button" class="btn green" id="sl-submit">Add to shortlist</button>
|
||||
<button type="button" class="btn ghost" data-close="modal-shortlist">Cancel</button>
|
||||
</div>
|
||||
<div id="sl-result"></div>
|
||||
</form>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="modal-backdrop" id="modal-search">
|
||||
<div class="modal">
|
||||
<h3>Save a search</h3>
|
||||
<form class="form" onsubmit="return false">
|
||||
<label>Search name</label>
|
||||
<input type="text" id="sv-name" placeholder="e.g. Available Toledo welders with OSHA">
|
||||
<label>SQL (will run against the catalog via /query/sql)</label>
|
||||
<textarea id="sv-sql" placeholder="SELECT * FROM workers_500k WHERE role='Welder' AND city='Toledo' AND state='OH' AND CAST(availability AS DOUBLE) > 0.5 LIMIT 20"></textarea>
|
||||
<div class="actions">
|
||||
<button type="button" class="btn green" id="sv-submit">Save search</button>
|
||||
<button type="button" class="btn ghost" data-close="modal-search">Cancel</button>
|
||||
</div>
|
||||
<div id="sv-result"></div>
|
||||
</form>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="modal-backdrop" id="modal-activity">
|
||||
<div class="modal">
|
||||
<h3>Log activity</h3>
|
||||
<form class="form" onsubmit="return false">
|
||||
<label>Action</label>
|
||||
<input type="text" id="act-action" placeholder="e.g. called, emailed, shortlisted, rejected">
|
||||
<label>Detail</label>
|
||||
<textarea id="act-detail" placeholder="What happened and what's the next step"></textarea>
|
||||
<div class="actions">
|
||||
<button type="button" class="btn green" id="act-submit">Log activity</button>
|
||||
<button type="button" class="btn ghost" data-close="modal-activity">Cancel</button>
|
||||
</div>
|
||||
<div id="act-result"></div>
|
||||
</form>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="footer">Lakehouse · workspaces · per-contract state, zero-copy handoff (Phase 8.5)</div>
|
||||
|
||||
<script>
|
||||
var P=location.pathname.indexOf('/lakehouse')>=0?'/lakehouse':'';
|
||||
var A=location.origin+P;
|
||||
|
||||
var state={list:[], current:null, currentAgent:'sarah'};
|
||||
|
||||
function el(t,c,x){var e=document.createElement(t);if(c)e.className=c;if(x!==undefined&&x!==null)e.textContent=String(x);return e}
|
||||
|
||||
function apiGet(p){
|
||||
return fetch(A+'/api'+p).then(function(r){return r.ok?r.json():r.text().then(function(b){throw new Error('HTTP '+r.status+': '+b)})});
|
||||
}
|
||||
function apiPost(p,body){
|
||||
return fetch(A+'/api'+p,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(body||{})}).then(function(r){
|
||||
return r.text().then(function(t){
|
||||
if(!r.ok) throw new Error('HTTP '+r.status+': '+t.slice(0,200));
|
||||
try{return JSON.parse(t)}catch(e){return {raw:t}}
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
function showModal(id){ document.getElementById(id).classList.add('show') }
|
||||
function hideModal(id){ document.getElementById(id).classList.remove('show') }
|
||||
|
||||
function loadList(preselectId){
|
||||
apiGet('/workspaces').then(function(arr){
|
||||
if(!Array.isArray(arr)) arr=[];
|
||||
state.list=arr;
|
||||
renderList();
|
||||
if(preselectId){ openWorkspace(preselectId); }
|
||||
else if(arr.length>0 && !state.current){ openWorkspace(arr[0].id); }
|
||||
}).catch(function(e){
|
||||
var host=document.getElementById('ws-list');host.textContent='';
|
||||
host.appendChild(el('div','empty','Failed to load: '+e.message));
|
||||
});
|
||||
}
|
||||
|
||||
function renderList(){
|
||||
var host=document.getElementById('ws-list');host.textContent='';
|
||||
if(state.list.length===0){
|
||||
host.appendChild(el('div','empty','No workspaces yet. Create one to get started.'));
|
||||
return;
|
||||
}
|
||||
state.list.slice().sort(function(a,b){
|
||||
return (b.updated_at||'').localeCompare(a.updated_at||'');
|
||||
}).forEach(function(w){
|
||||
var card=el('div','ws-card'+(state.current&&state.current.id===w.id?' active':''));
|
||||
card.onclick=function(){ openWorkspace(w.id); };
|
||||
card.appendChild(el('div','name',w.name||'(unnamed)'));
|
||||
var meta=el('div','meta');
|
||||
meta.appendChild(el('span','tier '+(w.tier||'weekly'),(w.tier||'weekly').toUpperCase()));
|
||||
meta.appendChild(document.createTextNode((w.owner||'?')+' · '+((w.shortlist||[]).length)+' shortlisted · '+((w.activity||[]).length)+' events'));
|
||||
card.appendChild(meta);
|
||||
host.appendChild(card);
|
||||
});
|
||||
}
|
||||
|
||||
function openWorkspace(id){
|
||||
apiGet('/workspaces/'+encodeURIComponent(id)).then(function(w){
|
||||
if(!w||w.error){ throw new Error(w&&w.error?w.error:'not found'); }
|
||||
state.current=w;
|
||||
renderList();
|
||||
renderDetail();
|
||||
}).catch(function(e){
|
||||
var host=document.getElementById('ws-detail');host.textContent='';
|
||||
host.appendChild(el('div','empty','Failed to load: '+e.message));
|
||||
});
|
||||
}
|
||||
|
||||
function renderDetail(){
|
||||
var host=document.getElementById('ws-detail');host.textContent='';
|
||||
var w=state.current;
|
||||
if(!w){ host.appendChild(el('div','empty','Select a workspace from the list, or create one.')); return; }
|
||||
|
||||
// Header
|
||||
var hdr=el('div','hdr-block');
|
||||
hdr.appendChild(el('div','wname',w.name||'(unnamed)'));
|
||||
var meta=el('div','wmeta');
|
||||
meta.appendChild(el('span','tier '+(w.tier||'weekly'),(w.tier||'weekly').toUpperCase()));
|
||||
meta.appendChild(document.createTextNode(' owner: '+(w.owner||'?')+' · created '+(w.created_at||'').slice(0,10)+' · updated '+(w.updated_at||'').slice(0,10)));
|
||||
hdr.appendChild(meta);
|
||||
if(w.description){ hdr.appendChild(el('div','desc',w.description)); }
|
||||
if((w.previous_owners||[]).length>0){
|
||||
var pos=el('div','prev-owners');
|
||||
pos.appendChild(el('span','chip','Previous owners:'));
|
||||
w.previous_owners.forEach(function(p){ pos.appendChild(el('span','chip',p)); });
|
||||
hdr.appendChild(pos);
|
||||
}
|
||||
// Action buttons
|
||||
var actions=el('div','actions');
|
||||
var handoffBtn=el('button','btn','Hand off');
|
||||
handoffBtn.onclick=function(){ showModal('modal-handoff'); };
|
||||
actions.appendChild(handoffBtn);
|
||||
var slBtn=el('button','btn ghost','Shortlist candidate');
|
||||
slBtn.onclick=function(){ showModal('modal-shortlist'); };
|
||||
actions.appendChild(slBtn);
|
||||
var svBtn=el('button','btn ghost','Save search');
|
||||
svBtn.onclick=function(){ showModal('modal-search'); };
|
||||
actions.appendChild(svBtn);
|
||||
var actBtn=el('button','btn ghost','Log activity');
|
||||
actBtn.onclick=function(){ showModal('modal-activity'); };
|
||||
actions.appendChild(actBtn);
|
||||
hdr.appendChild(actions);
|
||||
host.appendChild(hdr);
|
||||
|
||||
// Shortlist
|
||||
host.appendChild(sectionHeader('Shortlist',(w.shortlist||[]).length+' candidates'));
|
||||
if((w.shortlist||[]).length===0){
|
||||
host.appendChild(el('div','empty','No candidates shortlisted yet.'));
|
||||
} else {
|
||||
w.shortlist.forEach(function(sl){
|
||||
var e=el('div','entry');
|
||||
e.appendChild(el('div','title',(sl.dataset||'?')+' · '+(sl.record_id||'?')));
|
||||
if(sl.notes) e.appendChild(el('div','meta',sl.notes));
|
||||
e.appendChild(el('div','sub','by '+(sl.by_agent||sl.agent||'?')+' · '+(sl.at||sl.added_at||'').slice(0,16).replace('T',' ')));
|
||||
host.appendChild(e);
|
||||
});
|
||||
}
|
||||
|
||||
// Saved searches
|
||||
host.appendChild(sectionHeader('Saved searches',(w.saved_searches||[]).length+' queries'));
|
||||
if((w.saved_searches||[]).length===0){
|
||||
host.appendChild(el('div','empty','No saved searches yet.'));
|
||||
} else {
|
||||
w.saved_searches.forEach(function(sv){
|
||||
var e=el('div','entry');
|
||||
e.appendChild(el('div','title',sv.name||'(unnamed)'));
|
||||
if(sv.sql) e.appendChild(el('div','sub',sv.sql));
|
||||
e.appendChild(el('div','sub','by '+(sv.by_agent||sv.agent||'?')+' · '+(sv.at||sv.added_at||'').slice(0,16).replace('T',' ')));
|
||||
host.appendChild(e);
|
||||
});
|
||||
}
|
||||
|
||||
// Activity
|
||||
host.appendChild(sectionHeader('Activity',(w.activity||[]).length+' events'));
|
||||
if((w.activity||[]).length===0){
|
||||
host.appendChild(el('div','empty','No activity logged yet.'));
|
||||
} else {
|
||||
var act=el('div','activity');
|
||||
w.activity.slice().reverse().forEach(function(a){
|
||||
var e=el('div','entry');
|
||||
e.appendChild(el('div','title',a.action||'(no action)'));
|
||||
if(a.detail) e.appendChild(el('div','meta',a.detail));
|
||||
e.appendChild(el('div','sub','by '+(a.by_agent||a.agent||'?')+' · '+(a.at||'').slice(0,16).replace('T',' ')));
|
||||
act.appendChild(e);
|
||||
});
|
||||
host.appendChild(act);
|
||||
}
|
||||
}
|
||||
|
||||
function sectionHeader(title,count){
|
||||
var s=el('div','section-hdr');
|
||||
s.appendChild(el('span','title',title));
|
||||
s.appendChild(el('span','count',count));
|
||||
return s;
|
||||
}
|
||||
|
||||
function flash(id,text,kind){
|
||||
var box=document.getElementById(id);
|
||||
box.textContent='';
|
||||
box.appendChild(el('div','flash '+(kind||'ok'),text));
|
||||
}
|
||||
|
||||
function createWorkspace(){
|
||||
var name=document.getElementById('nw-name').value.trim();
|
||||
var desc=document.getElementById('nw-desc').value.trim();
|
||||
var tier=document.getElementById('nw-tier').value;
|
||||
var owner=document.getElementById('nw-owner').value.trim()||'sarah';
|
||||
if(!name){ flash('nw-result','Name is required','err');return; }
|
||||
apiPost('/workspaces/create',{name:name,description:desc,owner:owner,tier:tier}).then(function(r){
|
||||
state.currentAgent=owner;
|
||||
hideModal('modal-new');
|
||||
document.getElementById('nw-name').value='';
|
||||
document.getElementById('nw-desc').value='';
|
||||
loadList(r.id);
|
||||
}).catch(function(e){ flash('nw-result',e.message,'err'); });
|
||||
}
|
||||
|
||||
function submitHandoff(){
|
||||
var w=state.current;if(!w)return;
|
||||
var to=document.getElementById('ho-owner').value.trim();
|
||||
var reason=document.getElementById('ho-reason').value.trim();
|
||||
if(!to){ flash('ho-result','New owner required','err');return; }
|
||||
apiPost('/workspaces/'+encodeURIComponent(w.id)+'/handoff',{to_agent:to,reason:reason}).then(function(){
|
||||
state.currentAgent=to;
|
||||
hideModal('modal-handoff');
|
||||
loadList(w.id);
|
||||
}).catch(function(e){ flash('ho-result',e.message,'err'); });
|
||||
}
|
||||
|
||||
function submitShortlist(){
|
||||
var w=state.current;if(!w)return;
|
||||
var dataset=document.getElementById('sl-dataset').value.trim()||'workers_500k';
|
||||
var record=document.getElementById('sl-record').value.trim();
|
||||
var notes=document.getElementById('sl-notes').value.trim();
|
||||
if(!record){ flash('sl-result','Worker ID required','err');return; }
|
||||
apiPost('/workspaces/'+encodeURIComponent(w.id)+'/shortlist',{dataset:dataset,record_id:record,notes:notes,agent:state.currentAgent}).then(function(){
|
||||
hideModal('modal-shortlist');
|
||||
document.getElementById('sl-record').value='';
|
||||
document.getElementById('sl-notes').value='';
|
||||
loadList(w.id);
|
||||
}).catch(function(e){ flash('sl-result',e.message,'err'); });
|
||||
}
|
||||
|
||||
function submitSaveSearch(){
|
||||
var w=state.current;if(!w)return;
|
||||
var name=document.getElementById('sv-name').value.trim();
|
||||
var sql=document.getElementById('sv-sql').value.trim();
|
||||
if(!name||!sql){ flash('sv-result','Name and SQL are required','err');return; }
|
||||
apiPost('/workspaces/'+encodeURIComponent(w.id)+'/search',{name:name,sql:sql,agent:state.currentAgent}).then(function(){
|
||||
hideModal('modal-search');
|
||||
document.getElementById('sv-name').value='';
|
||||
document.getElementById('sv-sql').value='';
|
||||
loadList(w.id);
|
||||
}).catch(function(e){ flash('sv-result',e.message,'err'); });
|
||||
}
|
||||
|
||||
function submitActivity(){
|
||||
var w=state.current;if(!w)return;
|
||||
var action=document.getElementById('act-action').value.trim();
|
||||
var detail=document.getElementById('act-detail').value.trim();
|
||||
if(!action){ flash('act-result','Action required','err');return; }
|
||||
apiPost('/workspaces/'+encodeURIComponent(w.id)+'/activity',{action:action,detail:detail,agent:state.currentAgent}).then(function(){
|
||||
hideModal('modal-activity');
|
||||
document.getElementById('act-action').value='';
|
||||
document.getElementById('act-detail').value='';
|
||||
loadList(w.id);
|
||||
}).catch(function(e){ flash('act-result',e.message,'err'); });
|
||||
}
|
||||
|
||||
document.addEventListener('DOMContentLoaded',function(){
|
||||
document.getElementById('new-ws-btn').addEventListener('click',function(){ showModal('modal-new'); });
|
||||
document.getElementById('nw-create').addEventListener('click',createWorkspace);
|
||||
document.getElementById('ho-submit').addEventListener('click',submitHandoff);
|
||||
document.getElementById('sl-submit').addEventListener('click',submitShortlist);
|
||||
document.getElementById('sv-submit').addEventListener('click',submitSaveSearch);
|
||||
document.getElementById('act-submit').addEventListener('click',submitActivity);
|
||||
Array.prototype.forEach.call(document.querySelectorAll('[data-close]'),function(b){
|
||||
b.addEventListener('click',function(){ hideModal(b.getAttribute('data-close')); });
|
||||
});
|
||||
// Click-outside-to-close
|
||||
Array.prototype.forEach.call(document.querySelectorAll('.modal-backdrop'),function(m){
|
||||
m.addEventListener('click',function(e){ if(e.target===m) m.classList.remove('show'); });
|
||||
});
|
||||
loadList();
|
||||
});
|
||||
</script>
|
||||
</body></html>
|
||||
Reference in New Issue
Block a user