profit/lakehouse
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Compare commits

base: profit/lakehouse:demo-latest
Branches Tags
profit/lakehouse:main profit/lakehouse:demo/post-pr11-polish-2026-04-28 profit/lakehouse:infra/replace-gpt-oss-2026-04-28 profit/lakehouse:scrum/auto-apply-19814 profit/lakehouse:scrum/iter5-prep-2026-04-24 profit/lakehouse:test/enrich-prd-pipeline profit/lakehouse:cohesion/python-inventory-and-integration profit/lakehouse:phase/45-slice-4 profit/lakehouse:phase/45-slice-3 profit/lakehouse:ops/auditor-systemd-units profit/lakehouse:fix/upsert-outcome-update-merge profit/lakehouse:auditor/scaffold profit/lakehouse:fix/upsert-outcome-serde profit/lakehouse:federation-hnsw-trials profit/lakehouse:observer-kb
profit/lakehouse:demo-latest profit/lakehouse:distillation-v1.0.0 profit/lakehouse:v0.26-memory-complete
..
compare: profit/lakehouse:150cc3b6814b92d74385d546cbe762a4df1dab2f
Branches Tags
profit/lakehouse:demo/post-pr11-polish-2026-04-28 profit/lakehouse:main profit/lakehouse:infra/replace-gpt-oss-2026-04-28 profit/lakehouse:scrum/auto-apply-19814 profit/lakehouse:scrum/iter5-prep-2026-04-24 profit/lakehouse:test/enrich-prd-pipeline profit/lakehouse:cohesion/python-inventory-and-integration profit/lakehouse:phase/45-slice-4 profit/lakehouse:phase/45-slice-3 profit/lakehouse:ops/auditor-systemd-units profit/lakehouse:fix/upsert-outcome-update-merge profit/lakehouse:auditor/scaffold profit/lakehouse:fix/upsert-outcome-serde profit/lakehouse:federation-hnsw-trials profit/lakehouse:observer-kb
profit/lakehouse:demo-latest profit/lakehouse:distillation-v1.0.0 profit/lakehouse:v0.26-memory-complete
30 Commits
demo-latest .. 150cc3b681
Author SHA1 Message Date
root 150cc3b681 aibridge: LRU embed cache - 236x RPS gain on warm workloads. Per architecture_comparison.md universal-win for Rust side. Cache key (model,text), default 4096 entries, in-process inside gateway. Load test: 128 RPS -> 30k+ RPS, p50 78ms -> 129us.
lakehouse/auditor 20 blocking issues: cloud: claim not backed — "Verified end-to-end against persistent Go stack on :4110:"
2026-05-01 04:45:20 -05:00
rootandClaude Opus 4.7 9eed982f1a mcp-server: /_go/* pass-through for G5 cutover slice 
Adds an opt-in pass-through that routes Bun mcp-server requests
to the Go gateway when GO_LAKEHOUSE_URL is set. /_go/v1/embed,
/_go/v1/matrix/search etc. flow through Bun frontend → Go
backend without touching any existing tool. Off-by-default
(empty GO_LAKEHOUSE_URL → 503 with rationale); enabled via
systemd drop-in at:
  /etc/systemd/system/lakehouse-agent.service.d/go-cutover.conf

This is the first slice of real Bun-fronted traffic hitting the
Go substrate. The /api/* pass-through (Rust gateway) and every
existing tool are unmodified — fully additive cutover step.

Reversible: unset GO_LAKEHOUSE_URL or remove the systemd drop-in
and restart lakehouse-agent.service.

Verified end-to-end against persistent Go stack on :4110:
  /_go/health → {"status":"ok","service":"gateway"}
  /_go/v1/embed → nomic-embed-text-v2-moe vectors (dim=768)
  /_go/v1/matrix/search → 3/3 Forklift Operators (role+geo match)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-01 03:44:10 -05:00
rootandClaude Opus 4.7 3d068681f5 distillation: regenerated acceptance + audit reports (run_hash refresh)
lakehouse/auditor 17 blocking issues: cloud: claim not backed — "Verified end-to-end via playwright on devop.live/lakehouse:" 
Phase 6 acceptance + Phase 8 full-audit reports re-run; bit-for-bit
reproducibility property still holds (run 1 hash == run 2 hash),
just at a new value.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-30 00:13:17 -05:00
rootandClaude Opus 4.7 8de94eba08 cleanup: bump qwen2.5 → qwen3.5:latest in active defaults
lakehouse/auditor 16 blocking issues: cloud: claim not backed — "Verified end-to-end via playwright on devop.live/lakehouse:" 
stronger local rung is now the small-model-pipeline tier-1 default
across both Rust legacy + Go rewrite (cf. golangLAKEHOUSE phase 1).
same JSON-clean property as qwen2.5, more capacity. ollama still
serves both side-by-side; rollback is a 4-line revert if a workload
regresses.

active-default sites:
- lakehouse.toml [ai] gen_model + rerank_model → qwen3.5:latest
- mcp-server/observer.ts diagnose call (Phase 44 /v1/chat path) → qwen3.5:latest
- mcp-server/index.ts model roster doc → qwen3.5:latest first
- crates/vectord/src/rag.rs ContinuableOpts + RagResponse.model → qwen3.5:latest

skipped: execution_loop/mod.rs comments describing historic qwen2.5
tool_call quirks — those are documentation of past behavior, not
active defaults. data/_catalog/profiles/*.json are runtime-generated
(gitignored), not in scope for tracked changes.

cargo check -p vectord: clean. no behavioral change in the audit
pipeline — same JSON-clean local model, same think=Some(false)
posture, just stronger upstream.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-30 00:10:57 -05:00
rootandClaude Opus 4.7 d475fc7fff infra: replace gpt-oss with Ollama Pro + OpenCode Zen across hot paths 
Ollama Pro plan went live today (39-model fleet on the same
OLLAMA_CLOUD_KEY) and OpenCode Zen was already wired in the gateway
but not consumed. Routing every gpt-oss call site to faster /
stronger replacements:

| Site | gpt-oss → replacement | Why |
|---|---|---|
| ollama_cloud default | gpt-oss:120b → deepseek-v3.2 | newest DeepSeek revision; live-probed `pong` |
| openrouter default | openai/gpt-oss-120b:free → x-ai/grok-4.1-fast | already the scrum LADDER's PRIMARY |
| modes.toml staffing_inference | openai/gpt-oss-120b:free → kimi-k2.6 | coding-specialized, on Ollama Pro |
| modes.toml doc_drift_check | gpt-oss:120b → gemini-3-flash-preview | speed leader for factual checks |
| scrum_master_pipeline tree-split MAP+REDUCE | gpt-oss:120b → gemini-3-flash-preview | latency-dominated path (5-20× per file) |
| bot/propose.ts CLOUD_MODEL | gpt-oss:120b → deepseek-v3.2 | same Ollama key, faster |
| mcp-server/observer.ts overseer label fallback | gpt-oss:120b → claude-opus-4-7 | matches new overseer model |
| crates/gateway/src/execution_loop overseer escalation | ollama_cloud/gpt-oss:120b → opencode/claude-opus-4-7 | frontier reasoning matters here — fires only after local self-correct fails twice; Zen pay-per-token cost is bounded |

Verification:
- `cargo check -p gateway --tests` — clean
- Live probes through localhost:3100/v1/chat:
  - `opencode/claude-opus-4-7` → "pong"
  - `gemini-3-flash-preview` (ollama_cloud) → "pong"
  - `kimi-k2.6` (ollama_cloud) → "pong"
  - `deepseek-v3.2` (ollama_cloud) → "Pong! 🏓"

Notes:
- kimi-k2:1t still upstream-broken (HTTP 500 on Ollama Pro probe today,
  matches yesterday's memory). Replacement table never picks it.
- The Rust changes need a `systemctl restart lakehouse.service` to
  take effect on the running gateway. TS callers reload on next run.
- aibridge/src/context.rs still has gpt-oss:{20b,120b} in its window-
  size lookup table; harmless and kept for callers that pass it
  explicitly as an override.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-28 06:13:30 -05:00
rootandClaude Opus 4.7 f4dc1b29e3 demo: search.html — Live Market explainer rewrite + fp-bar viewport-paint + compact contract cards
lakehouse/auditor 18 blocking issues: cloud: claim not backed — "Verified end-to-end via playwright on devop.live/lakehouse:" 
Four UI changes landing together since they all polish Section ① and
Section ② of the public demo:

1. Section ① (Live Market — Chicago) explainer rewritten data-source-
   first ("Live from City of Chicago Open Data...") with bolded dial
   names so a skimmer can map the visual to the prose. Drops the
   "internal calendar" jargon and the slightly-overclaiming "rest of
   the page is reacting" framing — downstream sections read the same
   feed but don't react to the per-shift filter, so the new copy says
   "this row is its heartbeat" instead.

2. Fill-probability bar gets a left-to-right paint reveal (clip-path
   inset animation) so the green→gold→orange→red gradient reads as a
   *timeline growing* instead of a static heatmap with a "danger zone"
   at the right. Followed by a 30%-wide shimmer sweep on a 3.4s loop
   for live-signal feel.

3. Paint trigger moved from on-render to IntersectionObserver — by
   the time the user scrolls to Section ② the on-render animation had
   already finished. Now each bar paints in over 2.8s when it enters
   viewport (threshold 0.2, 350ms entry delay). Single shared observer,
   unobserve()s after firing so the watch list trends to zero.

4. Contract cards now compact-by-default with click-to-expand. New
   summary strip shows revenue / margin / fill-by-1wk / top candidate
   so scanners get the punchline without expanding. Click anywhere on
   the card surface (excluding inner content) to expand the full FP
   curve, economics grid, candidates list, and Project Index. Project
   Index auto-opens with the parent card so users actually find the
   build signals — but only on user-driven expand (avoiding 20× OSHA
   scrapes on page load). grid-template-rows: 0fr → 1fr animation
   handles the smooth height transition.

All four animations honor prefers-reduced-motion.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-28 06:01:04 -05:00
rootandClaude Opus 4.7 f892230699 demo: search.html UX polish — skeleton loader, card-in stagger, hero takeover, B&W faces 
Search results no longer pop in as a single block. New behavior:

- Skeleton list pre-claims the vertical space results will occupy
  with shimmering placeholder cards, so arriving results fade in
  over the skeleton instead of pushing layout. Sweep is staggered
  per row for a "rolling wave" not "everything blinking together".
- Domain-language stage caption ("matching against permits",
  "ranking by reliability") rotates on a fixed schedule so users
  read progress, not a stuck spinner.
- @keyframes card-in: real worker cards rise 4px and fade in over
  350ms with nth-child stagger across the first ~12 rows. Honors
  prefers-reduced-motion.
- Avatar imgs filter through grayscale + slight contrast/blur to
  pull the SDXL Turbo color cast (which screams "AI generated" at
  small sizes). Cert icons get the same treatment.
- Once-per-session hero takeover compresses the Section ⓪ strip
  ("Not a CRM — an index that learns from you") into a centered
  hero on first paint, dismissed by clicking anywhere. Stats
  hydrate from live endpoints.

console.html: mirrors the avatar B&W filter for visual consistency,
and removes the headshot insertion entirely — back to monogram
initials. The console (internal staffer view) doesn't need synthetic
faces; the public demo at /lakehouse/ does.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-28 06:01:04 -05:00
rootandClaude Opus 4.7 4b92d1da91 demo: icon recipe pipeline + role-aware portraits + ComfyUI negative-prompt override 
Adds two single-source-of-truth recipe files that drive both the
hot-path render server and the offline pre-render scripts:

- role_scenes.ts: per-role-band scene clauses (clothing + backdrop).
  Forklift operators look like forklift operators instead of
  collapsing to interchangeable studio shots. SCENES_VERSION mixes
  into the headshot cache key so a coordinator tweak refreshes every
  matching face on next view.
- icon_recipes.ts: cert / role-prop / status / hazard / empty icons
  with deterministic per-recipe seeds + fuzzy text resolver.
  ICONS_VERSION suffix on the cached file means edits don't
  overwrite in place — misfires are recoverable.

Routes (mcp-server/index.ts):
- GET /headshots/_scenes — exposes SCENES + version to the
  pre-render script so prompts don't drift between batch and hot-path.
- GET /icons/_recipes — same idea for icons.
- GET /icons/cert?text=... — resolves free-text cert names to a
  recipe and 302s to the rendered icon. 404 (not 500) when no recipe
  matches so the front-end can hang `onerror="this.remove()"`.
- GET /icons/render/{category}/{slug} — cache-or-render at 256² (8
  steps) for crisper edges than 512² when downsampled to 14px.

ComfyUI portrait support (scripts/serve_imagegen.py):
The editorial workflow had `human, person, face` baked into its
negative prompt — actively sabotaging portraits. _comfyui_generate
now accepts negative_prompt/cfg/sampler/scheduler overrides, and
those mix into the cache key so portrait calls don't collapse into
hero-shot cache hits.

scripts/staffing/render_role_pool.py: pre-renders the role-aware
face pool by reading SCENES from /headshots/_scenes — single source
of truth verified at run time.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-28 06:01:04 -05:00
rootandClaude Opus 4.7 1745881426 staffing: face pool fetch preserves prior tags + --shrink gate + atomic manifest write 
fetch_face_pool was wiping 952 hand-classified rows when re-run from
a Python without deepface installed (it reset every gender to None).
Now:

- Loads existing manifest by id and overlays only fetch-owned fields,
  so gender/race/age/excluded survive a refetch.
- deepface pass tags only records that don't already have a gender;
  deepface unavailable means "leave existing tags alone" not "reset".
- New --shrink flag required to drop ids >= --count. Default refuses
  to shrink the pool silently.
- Atomic write via tmp + os.replace so an interrupted run can't
  corrupt the manifest.
- Dedupes duplicate id lines (root cause of the 2497-row manifest
  backing a 1000-face pool).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-28 06:01:04 -05:00
rootandClaude Opus 4.7 a05174d2fa ops: track tif_polygons.ts orphan import 
entity.ts imports findTifDistrict from ./tif_polygons.js but the
source file was never committed — only present in the working tree.
Adding it so a fresh clone compiles.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-28 06:01:04 -05:00
rootandClaude Opus 4.7 f9a408e4c4 Surname → ethnicity routing + ComfyUI fallback for sparse pool buckets + cache-buster 
Three problems J flagged ("not matching properly", "same faces", "still
showing old icons") had three different roots:

1. MISMATCH: front-end was first-name only, so "Anna Cruz" / "Patricia
   Garcia" / "John Jimenez" all defaulted to caucasian. Added
   SURNAMES_HISPANIC / _SOUTH_ASIAN / _EAST_ASIAN / _MIDDLE_EASTERN
   dicts to both search.html and console.html. Surname is checked
   FIRST (stronger signal for hispanic + asian than first names),
   then first-name fallback. Cruz → hispanic, Patel → south_asian,
   Nguyen → east_asian, regardless of first name.

2. SAME FACES: pool buckets are uneven — woman/south_asian=3,
   man/black=4, woman/middle_eastern=2 — so any worker in those
   buckets collapses to 2-4 photos no matter how good the hash is.
   /headshots/:key now 302-redirects to /headshots/generate/:key
   when the gender × race intersection is below 30 faces. ComfyUI
   on-demand gives infinite uniqueness for the sparse buckets
   (deterministic-per-worker via djb2 seed). Dense buckets still
   serve from the pool — no GPU cost there.

3. STALE CACHE: Cache-Control was max-age=86400, immutable — pinned
   old photos in browsers for 24h after any server-side update.
   Dropped to max-age=3600, must-revalidate, and added a v=2
   cache-buster query param to all front-end /headshots/ URLs so
   existing cached entries are bypassed on next page load.

Also surfacing X-Face-Pool-Bucket / Bucket-Size headers for diagnosis.

Verified: playwright run shows surname routing correct (Torres,
Rivera, Alvarez, Gutierrez, Patel, Nguyen, Omar all bucketed
correctly), sparse buckets 302 to ComfyUI, dense buckets stay on
the thumb pool.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-28 06:01:04 -05:00
rootandClaude Opus 4.7 a3b65f314e Synthetic face pool — 1000 StyleGAN headshots, ComfyUI hot-swap, 60x smaller thumbs 
Worker cards now ship a real photo per person instead of monogram tiles:

  - fetch_face_pool.py pulls 1000 faces from thispersondoesnotexist.com
  - tag_face_pool.py runs deepface for gender/race/age, excludes <22yo
  - manifest.jsonl: 952 servable, gender/race buckets populated
  - /headshots/_thumbs/ pre-resized to 384px webp (587KB -> 11KB,
    60x smaller; without this Chrome's parallel-connection budget
    drops ~75% of tiles in a 40-card grid)
  - /headshots/:key gender x race x age intersection bucketing with
    gender-only fallback when intersection is sparse
  - /headshots/generate/:key ComfyUI on-demand for the contractor
    profile spotlight (cold ~1.5s, cached ~1ms; worker-derived
    djb2 seed makes faces deterministic-per-worker but unique
    across workers sharing the same prompt)
  - serve_imagegen.py _cache_key() now includes seed (was caching
    by prompt only -> 3 different worker seeds collapsed to 1
    cached image; verified fix produces 3 distinct md5s)
  - confidence-default name resolution: Xavier->man+hispanic,
    Aisha->woman+black, etc. Every worker resolves to a bucket.

End-to-end: playwright run on /?q=forklift+operators+IL -> 21/21
cards loaded, 0 broken, all 384px webp.

Cache + binary pool gitignored; manifest tracked.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-28 06:01:04 -05:00
root 10ed3bc630 demo: real synthetic headshots — fetch pool + serve route + UI wire 
Three layers shipped:

1. SCRIPT — scripts/staffing/fetch_face_pool.py
   Pulls N synthetic StyleGAN faces from thispersondoesnotexist.com
   into data/headshots/face_NNNN.jpg, writes manifest.jsonl. Idempotent:
   re-running skips existing files. Optional gender tagging via deepface
   (currently unavailable on this box; the script handles ImportError
   gracefully and tags everything as untagged). Fetched 198 faces with
   concurrency=3 in ~67s.

2. SERVER — /headshots/:key route in mcp-server/index.ts
   Loads manifest at first hit, caches in globalThis._faces. Hashes the
   key with djb2-style mixing → pool index → returns the JPG. Same
   key always gets the same face (deterministic). Accepts
   ?g=man|woman&e=caucasian|black|hispanic|south_asian|east_asian|middle_eastern
   to bias pool selection — the gender/ethnicity buckets fall back to
   the full pool when no tagged matches exist. Cache-Control:
   86400 immutable so faces ride the browser cache after first hit.
   /headshots/__reload re-reads the manifest without restart.

3. UI — search.html + console.html worker cards
   Re-added overlay <img> on top of the monogram .av circle. img.src
   = /headshots/<encoded-key>?g=<hint>&e=<hint>. img.onerror removes
   the failed image so the monogram stays visible if the face pool
   isn't fetched / CDN is blocked. .av now has overflow:hidden +
   position:relative to clip the img to a perfect circle.

Forced-confident name resolution (J: "we're CREATING the profile,
created as though you truly have the information Xavier is more
likely Hispanic and he's a male"):

   genderFor(name)        — looks up MALE_NAMES + FEMALE_NAMES,
                            falls back to a deterministic hash split
                            so unknown names spread ~50/50. Sets now
                            include cross-cultural names: Alejandro/
                            Andres/Mateo/Santiago/Joaquin/Cesar/Hugo/
                            Felipe/Gerardo/Salvador/Ramon (Hispanic),
                            Raj/Anil/Vikram/Krishna/Pradeep (South
                            Asian), Wei/Yi/Hiroshi/Akira/Hyun (East
                            Asian), Demetrius/Kareem/DaQuan/Khalil
                            (Black), Omar/Khalid/Hassan/Ahmed/Bilal
                            (Middle Eastern). FEMALE_NAMES extended
                            in parallel.

   guessEthnicityFromFirstName(name)
                          — confident default of 'caucasian' for any
                            name not in the cultural buckets so every
                            worker resolves to a category the face
                            pool can be biased toward. Order: ME → Black
                            → Hispanic → South Asian → East Asian →
                            Caucasian (matters where names overlap,
                            e.g. Aisha appears in ME + Black, biases
                            toward ME for visual fit).

   Both helpers also ported into console.html so the triage backfills
   and try-it-yourself rendering get the same hint stack.

Privacy note in the script + route comments: the synthetic data uses
the worker's name as the seed; production should hash worker_id (not
name) to avoid leaking PII to a third-party CDN. The fetch URL itself
is referenced once per pool build, not per-worker.

.gitignore — added data/headshots/face_*.jpg (~100MB for 198 faces;
the manifest + script are tracked). Re-running the script on a fresh
checkout rebuilds the pool from scratch.

Verified end-to-end via playwright on devop.live/lakehouse:
   forklift query → 10 worker cards
   10/10 with face images (real synthetic headshots, not monograms)
   0/10 broken
   Alejandro G. Nelson  → ?g=man&e=hispanic
   Patricia K. Garcia    → ?g=woman&e=caucasian
   Each name → unique face, deterministic across loads.
   Console triage backfills get the same treatment.
 2026-04-28 06:01:04 -05:00
root cdf5f5926a demo: console — sober worker cards (mirror dashboard styling) 
J: "can you update Staffer's Console too the same look." Console
rendered worker rows in three places (Chapter 4 permit-contract
candidates, Chapter 8 triage backfills, Chapter 9 try-it-yourself
results) with the original 28px square avatar + flat backgrounds —
inconsistent with the new dashboard design.

Three changes:

1. CSS — .worker now has a 3px left-edge border that color-codes the
   role family, and .av is a 32px circle with a muted dark background
   + 1px ring + monogram initials. Five role-band colors mirror
   search.html: warehouse blue / production amber / trades purple /
   driver green / lead orange. Plus a .role-pill style matching the
   dashboard's small uppercase chip.

2. Helpers — added ROLE_BANDS regex table + roleBand() classifier and
   a new workerRow(name, role, detail, opts) builder. Same regex
   patterns as search.html so a "Forklift Operator" classifies
   identically on every page. opts.endorsed adds the green endorsed
   chip; opts.score appends a rank badge.

3. Replaced the three inline avatar+row constructors with workerRow()
   calls. Net: console.html lost ~20 lines of duplicated DOM building
   while gaining role bands + pills.

Verified end-to-end via playwright on devop.live/lakehouse/console:
  Chapter 8 triage scenario "Marcus running late site 4422":
    5 backfill rows render with [warehouse] band + WAREHOUSE pill +
    monogram avatars (SBC, ETW, SHC, WMG, MEB).
  Same sober look as the dashboard worker cards. No emojis, no
  cartoons, color-coded role family on the left edge.
 2026-04-28 06:01:04 -05:00
root f92b55615f demo: worker cards — sober monogram avatars + role bands (no cartoons) 
J: "It's two cartoonish right now the website looks like it was made
by first grade teacher." Pulled the DiceBear personas-style headshots
and the emoji role badges. They were generative-illustration playful;
this is supposed to read like a staffing tool, not a kindergarten
attendance sheet.

Replacement design — restraint, signal, no glyphs:

  Avatar:   40px circle, monogram initials, muted dark background
            (#161b22), 1px ring (#21262d), white-ish text. No image,
            no emoji. Looks like a pre-photo placeholder slot in a
            real ATS.

  Role band: the role gets classified into one of five families:
            WAREHOUSE / PRODUCTION / SKILLED TRADE / DRIVER / LEAD
            (regex-based; falls back to the first word of the role
            for unknown families). Each family has a single muted
            color: blue / amber / purple / green / orange. The
            color appears as:
              - a 3px left border on the .iworker card
              - a 2px left border + matching text color on a small
                uppercase pill in the detail line

That's it. No images, no emojis, no per-role illustrations. The
staffer sees role-family at a glance via the band color, name and
initials prominently, full role + city + zip in the detail string
behind the pill. Five colors total instead of an eight-color rainbow.

CSS:
  .iworker[data-role-band="warehouse"] etc. → 3px left border
  .role-pill[data-rb="warehouse"] etc.      → matching pill border

JS:
  ROLE_BANDS = 6 regex → band+label entries (warehouse, production,
                          trades, driver, lead, quality)
  roleBand(role)       = first matching entry, fallback to first
                          word of role uppercased

Verified end-to-end via playwright on devop.live/lakehouse:
  forklift query → 10 cards
  every card → monogram avatar + WAREHOUSE pill (blue band)
  no images, no emojis, no rainbow

Restart sequence after these edits:
  pkill -9 -f "/home/profit/lakehouse/mcp-server/index.ts"
  ( setsid bun run /home/profit/lakehouse/mcp-server/index.ts \
      > /tmp/mcp-server.log 2>&1 < /dev/null & disown )
 2026-04-28 06:01:04 -05:00
root d571d62e9b demo: spec — refresh repo layout + model fleet + per-staffer + paths 8-9 
J: "how about devop.live/lakehouse/spec." Spec was anchored on
2026-04-21 state (v2 footer): mistral mentioned in the model matrix,
13 crates not 15, missing validator/truth/auditor crates, no mention
of OpenCode 40-model fleet, no pathway memory, no per-staffer
hot-swap, no Construction Activity Signal Engine, ADR count was 20.
Footer claimed Phases 19-25.

Edits, in order:

  Ch1 Repository layout
    + crates/truth/ (ADR-021 rule store)
    + crates/validator/ (Phase 43 — schema/completeness/policy gates)
    + auditor/ (cross-lineage Kimi↔Haiku/Opus auto-promote)
    + scripts/distillation/ (frozen substrate v1.0.0 at e7636f2)
    Updated aibridge to mention ProviderAdapter dispatch
    Updated gateway to mention OpenAI-compat /v1/* drop-in middleware
    Updated mcp-server route list to include /staffers + profiler/contractor pages
    Updated config/ to mention modes.toml + providers.toml + routing.toml
    Updated docs/ ADR count from 20 → 21
    Updated data/ to mention _pathway_memory + _auditor/kimi_verdicts

  Ch3 Measurement & indexing
    REPLACED stale "Model matrix (Phase 20)" T1-T5 table that
    mentioned mistral with the current 5-provider fleet:
      ollama / ollama_cloud / openrouter / opencode (40 models, one
      sk-* key reaches Claude Opus 4.7, GPT-5.5-pro, Gemini 3.1-pro,
      Kimi K2.6, GLM, DeepSeek, Qwen, MiniMax, free) / kimi
    ADDED 9-rung cloud-first ladder pseudocode
    ADDED N=3 consensus + cross-architecture tie-breaker math
    ADDED auditor cross-lineage Kimi K2.6 ↔ Haiku 4.5 + Opus auto-promote
    ADDED distillation v1.0.0 freeze paragraph (145 tests, 22/22, 16/16)
    Updated Continuation primitive to mention Phase 44 Rust port

  Ch5 What a CRM can't do
    Extended the table with 6 new capabilities:
      - Per-staffer relevance gradient
      - Triage in one shot (late-worker → backfills + draft SMS)
      - Permit → fill plan derivation
      - Public-issuer attribution across contractor graph
      - Cross-lineage AI audit on every PR
      - Pathway memory (system-level hot-swap, ADR-021)

  Ch6 How it gets better over time
    Lede updated from 7 paths → 10 paths
    NEW Path 7 — Pathway memory (ADR-021)
    NEW Path 8 — Per-staffer hot-swap index
    NEW Path 9 — Construction Activity Signal Engine
    Original Path 7 (observer ingest) renumbered to Path 10

  Ch9 Per-staffer context
    Lede now anchors Path 8 from Ch6
    NEW lead section: Per-staffer hot-swap index — Maria/Devon/Aisha,
    same query reshapes per coordinator (167 IL / 89 IN / 16 WI),
    MARIA'S MEMORY pill, /staffers endpoint, metro-agnostic by
    construction. The original Phase 17 profile / Phase 23 competence
    sections retained beneath as the deeper architecture detail.

  Ch10 A day in the life
    Updated 14:00 emergency event to use the late-worker triage
    handler — coordinator types "Dave running late site 4422", gets
    profile + draft SMS + 5 backfills + Copy SMS button in 250ms.
    The old Click No-show button → /log_failure flow remains valid
    (penalty still records); the user-facing surface is the new
    triage card.

  Ch11 Known limits
    REPLACED the Mem0/Letta/Phase-26 era list with current honest
    limits: BAI persistence + backtesting, NYC DOB adapter, 12
    awaiting public-data sources for contractor profile, rate/margin
    awareness, Mem0-style UPDATE/DELETE, Letta hot cache (now 5K
    not 1.9K), confidence calibration, SEC fuzzy precision, tighter
    pathway+scrum integration.

  Footer
    v2 2026-04-21 → v3 2026-04-27
    Phases 19-25 → 19-45
    Lists today's phases: distillation v1.0.0 substrate, gateway as
    OpenAI-compat drop-in, mode runner, validator + iterate, ADR-021
    pathway memory, per-staffer hot-swap, Construction Activity Signal
    Engine.

  Nav
    + Profiler link
    Date pill v1 · 2026-04-20 → v3 · 2026-04-27

Verified end-to-end on devop.live/lakehouse/spec — 11 chapter h2s
render in order, 67KB page (was 50KB-ish), all internal links resolve.
 2026-04-28 06:01:04 -05:00
root 631b0329b1 demo: proof — full architecture-page rewrite for current state 
J: "needs a rewrite." Old version was anchored on a dual-agent
mistral+qwen2.5 loop that hasn't been the model story for weeks,
called the system 13 crates (it's 15), referenced "Local 7B models"
in the honest-limits section, and had no mention of:
  - the 40-model OpenCode fleet via one sk-* key
  - the 9-rung cloud-first ladder
  - N=3 consensus + cross-architecture tie-breaker
  - auditor cross-lineage (Kimi K2.6 ↔ Haiku 4.5, Opus auto-promote)
  - distillation v1.0.0 frozen substrate (e7636f2)
  - pathway memory (88 traces, 11/11 replays, ADR-021)
  - per-staffer hot-swap index
  - Construction Activity Signal Engine + BAI + ticker network
  - the gateway as OpenAI-compat drop-in middleware

Rewrote into 10 chapters:

  1.  Receipts — live tests + new live tile showing the Signal Engine
      view for THIS load (issuer count, attributed build value,
      contractor count, attribution edges)
  2.  Architecture — corrected to 15 crates with current responsibilities;
      ASCII diagram showing OpenAI consumers + MCP + Browser all hitting
      gateway /v1/*; provider fleet table with all 5 (ollama, ollama_cloud,
      openrouter, opencode 40-model, kimi); validator + truth + auditor
      crates added
  3.  Model fleet — REPLACED the dual-agent mistral story. Now: the
      9-rung ladder (kimi-k2:1t through openrouter:free → ollama local),
      N=3 consensus + tie-breaker math, auditor Kimi↔Haiku alternation
      with Opus auto-promote on big diffs, distillation v1.0.0 freeze
      tag e7636f2 (145 tests · 22/22 · 16/16 · bit-identical)
  4.  Two memory layers — kept playbook content (Phase 19 boost math
      still load-bearing), added pathway memory (ADR-021) section with
      live counters in the page (88 / 11-11 / 100% reuse rate)
  5.  Per-staffer hot-swap — NEW. Pseudocode showing how staffer_id
      scopes state filter + playbook geo + UI relabel to MARIA'S MEMORY
  6.  Construction Activity Signal Engine — NEW. Three attribution
      flavors (direct, parent, associated), BAI math, cross-metro
      replication framing (NYC DOB next, then LA / Houston / Boston)
  7.  Architectural choices — added ADR-021 row + distillation freeze row
  8.  Measured at scale — kept (uses /proof.json scale data)
  9.  Verify or dispute — REFRESHED with current endpoints. Removed the
      stale "bun run tests/multi-agent/scenario.ts" recipe; added curl
      examples for /v1/health, pathway/stats, per-staffer scoping (3-loop
      bash script), late-worker triage, profiler_index, ticker_quotes,
      auditor verdicts, distillation acceptance gate
  10. What we are NOT claiming — REFRESHED. Removed "Local 7B models"
      caveat; added: 12 awaiting public-data sources are placeholders,
      SEC name-fuzzy has rare false positives, BAI is a thesis not a
      backtest yet, single-metro today

Live data probes added:
  loadPathwayLive   — fills pwm-traces / pwm-replays / pwm-rate spans
  loadSignalLive    — renders the LIVE Signal Engine tile under Ch1

Nav also gained a Profiler link to match search.html and console.html.

Verified end-to-end on devop.live/lakehouse/proof:
  10 chapters render, 5/5 live tests pass, pathway shows 88 traces +
  100% reuse rate, live signal tile shows 11 issuers + $347M attributed
  + 200 contractors + 14 attribution edges. Architecture diagram +
  crate table accurate as of HEAD.
 2026-04-28 06:01:04 -05:00
root 4c46cf6a21 demo: console — three new chapters reflecting recent shipments 
J: "it's outdated." Console walkthrough was stuck on the original 6
chapters (legacy-bridge / permits / catalog / ranking demo / playbook
memory / try-it-yourself). Three weeks of new work weren't visible.

Three new chapters added between the existing playbook-memory chapter
and the input box; all pull live data from the running system:

  Chapter 6 — Three coordinators, three views of the same corpus
    Renders Maria/Devon/Aisha cards from /staffers with their
    territories. Frames the per-staffer hot-swap as the relevance
    gradient that compounds independently per coordinator. Same query
    "forklift operators" returns 89 IN / 16 WI / 167 IL workers
    depending on who's acting.

  Chapter 7 — The hidden signal — public issuers in your contractor graph
    Pulls /intelligence/profiler_index, builds the basket, shows
    issuer count + attributed build value + contractor count as the
    three top metrics. Lists top 8 issuers with attribution counts
    and direct-link to the profiler. This is the BAI / Signal Engine
    pitch in walkthrough form: every contractor name is also a forward
    indicator on a public equity. Cross-metro replication explicit
    in the closing paragraph.

  Chapter 8 — When something breaks — triage in one shot
    Live triage demo against /intelligence/chat with body
    {message:"Marcus running late site 4422"}. Renders the worker
    card + draft SMS + 5 backfills + duration_ms. The 250ms-vs-20min
    moment, made concrete with real Quincy IL workers.

Chapter 9 (was 6) — Try it yourself
  Updated input examples to demonstrate each new route:
    "8 production workers near 60607" → headcount + zip parser
    "Marcus running late site 4422"  → triage handler
    "Marcus"                          → bare-name lookup
    "what came in last night"         → temporal route
    "reliable forklift operators with OSHA certs" → hybrid SQL+vector
  Each is a click-to-run link beneath the input.

Two new accent classes: .accent-g (green for issuer-count) and
.accent-r (red for triage event).

Verified end-to-end on devop.live/lakehouse/console: 9 chapters
render, ch6 shows 3 staffer personas, ch7 shows 11 issuers / $347M /
200 contractors, ch8 shows Marcus V. Campbell + draft SMS + 5
backfills.
 2026-04-28 06:01:04 -05:00
root 6366487b45 ops: persist runtime fixes — iterate.rs unused state, catalog cleanup 
Two load-bearing runtime changes that were never committed:

1. crates/gateway/src/v1/iterate.rs — `state` → `_state` on the unused
   route-state parameter. Cleared the one cargo workspace warning.
   Fix was made earlier this session but the working-tree change
   never made it into a commit.

2. data/_catalog/manifests/564b00ae-cbf3-4efd-aa55-84cdb6d2b0b7.json —
   DELETED. This was the dead manifest for `client_workerskjkk`, a
   typo dataset whose parquet was deleted but whose catalog entry
   stayed registered. Every SQL query failed schema inference on the
   missing file before reaching its target table — that's the bug
   that made /system/summary report 0 workers and the demo show zero
   bench. Deleting the manifest keeps the fix on disk; committing
   the deletion keeps it in git so a fresh checkout doesn't regress.

3. data/_catalog/manifests/32ee74a0-59b4-4e5b-8edb-70c9347a4bf3.json
   — runtime catalog metadata update from the successful_playbooks_live
   write path. Ride-along change.

Reports under reports/distillation/phase[68]-*.md are auto-regenerated
by the audit cycle each run; skipping those.
 2026-04-28 06:01:04 -05:00
root db81fd8836 demo: System Activity panel — capability index reflects every recent shipment 
Old panel showed playbook ops + search counts and went empty in a
fresh demo (no operations yet). J: "update System Activity to coincide
with all of our recent updates."

Rebuilt as a live capability index — each tile is a thing the
substrate has learned to do, with the metric proving it's running.
Pulled in parallel from /staffers, /system/summary,
/api/vectors/playbook_memory/stats, /api/vectors/pathway/stats,
/intelligence/profiler_index, /intelligence/activity. Each probe
catches its own error so a single missing endpoint doesn't collapse
the panel.

Nine capability cards (verified end-to-end on devop.live/lakehouse):

  1. Per-staffer hot-swap index           3 personas (Maria/Devon/Aisha)
  2. Construction Activity Signal Engine  11 issuers · $347M attributed
                                          build value · network 11/14
  3. Late-worker / no-show triage         one-shot — name+late → backfills+SMS
  4. Permit → staffing bridge             24/day, every Chicago permit ≥$250K
  5. Hybrid SQL + vector search           500K workers · 5,474 playbook entries
  6. Schema-agnostic ingestion            36 datasets · 2.98M rows
  7. Contractor profile + project index   6 wired · 12 queued sources
  8. Pathway memory                       88 traces · 11/11 replays · 100%
  9. Ticker association network           11 tickers · 3 direct + 11 associated

Each card carries:
  - capability title + ship date pill ("baseline" or "shipped 2026-04-27")
  - big metric (live, not pre-baked)
  - sub-context line in coordinator language
  - "why a staffer cares" explanation
  - optional "Open →" deep link to the surface (Profiler, Contractor)

Header + intro paragraph reframed: "what the substrate has learned to
do" instead of "what the substrate has learned." Operational learning
(fills, playbooks, hot-swaps) compounds INSIDE each capability; the
panel surfaces the set of capabilities the corpus knows how to express.

Closing operational-stats row at the bottom shows fills/searches/
recent playbooks when /intelligence/activity has any.
 2026-04-28 06:01:04 -05:00
root a789000982 demo: profiler — Construction Activity Signal Engine narrative + BAI 
J's prompt: shoot for the stars, frame the data corpus's value as a
predictive signal, not just a contractor directory. The thesis is
that every name in this corpus is also a forward indicator on public
equities — permits filed today predict construction starts in ~45
days, staffing in ~30, revenue recognition months later. The
associated-ticker network surfaces this signal before any 10-Q does.

Two new layers above the basket:

1. HERO THESIS PANEL — "Chicago Construction Activity Signal Engine"
   header + 3-line value statement, then 4 live metrics:

   - BAI (Building Activity Index) — attribution-weighted average of
     day-change % across surfaced issuers. Weight = attribution count
     so issuers we have more depth on count more. Today: +0.76%
     (9 issuers · top contributors FCBC +2.4%, ACRE +1.7%, JPM +1.5%).
     Color-coded green/red.

   - Indexed build value — total $ of permits attributable to ANY
     public issuer in this view. Today: $344M.

   - Network depth — issuers / attribution edges. Today: 9 / 15.
     This is the "we see what nobody else sees" metric: how many
     contractors are bridges from a private builder back to a public
     equity holder.

   - Market replication roadmap — chips showing "Chicago — live ·
     NYC DOB — adapter ready · LA County · Houston BCD · Boston ISD
     · DC DCRA". Frames the corpus as metro-agnostic from day one.

2. PER-TICKER ACTIVITY MAP — when a basket card is clicked, a leaflet
   map appears below the basket plotting that ticker's geocoded permit
   activity. Pulls /intelligence/contractor_profile for up to 6
   attributed contractors, merges their geocoded permits, plots on a
   dark Chicago tile layer. Color-banded by permit cost (green <$100K,
   amber $100K-$1M, red ≥$1M). Click TGT → 23 Target permits across
   Chicago; click JPM → JPMorgan-adjacent contractor activity. Cached
   per ticker so toggling is instant.

Verified end-to-end on devop.live/lakehouse/profiler:
  Default load: hero panel renders with all 4 metrics, basket strip
                with 9 issuers + live prices in 669ms.
  Click TGT  : signal map activates, "23 geocoded permits across
                1 contractor", table filters to 2 rows.
  Tooltip on basket cards: full reason path including matched name +
                contributors attributed to that ticker.

Architecture-side: zero new server code — all metrics computed
client-side from the existing profiler_index + ticker_quotes payloads.
The corpus already had the value; the page just needed to articulate it.
 2026-04-28 06:01:04 -05:00
root aa56fbce61 demo: profiler — scrolling ticker basket with live prices + click-to-filter 
J asked: "kind of like a scrolling ticker that has all of the companies
and their stock prices and where they fit in the map." Implemented as
a horizontal-scroll strip at the top of /profiler:

  9 public issuers in this view · quotes via Stooq · 669ms
  ┌────┬────┬────┬────┬────┬────┬────┐
  │TGT │JPM │BALY│ACRE│FCBC│NREF│LSBK│ ← live price + day-change per
  │129 │311 │... │... │... │... │... │   ticker, color-banded by
  │+.17│+1.5│... │... │... │... │... │   attribution kind
  └────┴────┴────┴────┴────┴────┴────┘

Each card carries:
  - ticker + live price + day-change % (red/green)
  - attribution count + kind (exact / direct / parent / associated)
  - left bar color = strongest attribution kind (green for direct
    issuer, amber for parent, blue for co-permit associated, gradient
    when both direct and associated apply)
  - tooltip on hover lists the contractors attributed to this ticker
  - click toggles a filter on the table below — clicking TGT cuts the
    200-row list down to just TARGET CORPORATION + TORNOW, KYLE F
    (Target's primary co-permit contractor)

Server-side:
- entity.ts exports fetchStooqQuote (was internal)
- new POST /intelligence/ticker_quotes — accepts {tickers: [...]},
  fans out to Stooq.us in parallel, returns
  {ticker, price, price_date, open, high, low, day_change_pct,
   stooq_url} per symbol or null for non-US listings (HOC.DE, SKA-B.ST,
   LLC.AX). Capped at 50 symbols per call.

Front-end:
- mcp-server/profiler.html — new .basket-wrap section above the
  controls. buildBasket() runs after profiler_index loads:
    1. Aggregates unique tickers from .tickers.direct + .associated
       across all surfaced contractors
    2. Renders shells immediately (ticker symbol + "—" placeholder)
    3. Batch-fetches quotes via /intelligence/ticker_quotes
    4. Updates each card with price + day-change in place
  Click on a card sets a tickerFilter; render() skips rows whose
  attributions don't include that ticker. "clear filter" button on
  the basket strip resets it.

Verified end-to-end on devop.live/lakehouse/profiler:
  Default load → 9 issuers, live prices populated in 669ms
  TGT click   → table filters to TARGET CORPORATION + TORNOW, KYLE F
                (the contractor who runs 3 of Target's recent permits
                gets the TGT correlation indicator)
  JPM card    → $311.63, +1.55% — JPMorgan-adjacent contractors
  Tooltip     → list of contractors attributed to the ticker
 2026-04-28 06:01:04 -05:00
root ba41ad2846 demo: profiler index — ticker associations (direct, parent, co-permit) 
J's framing: "if a contractor works for Target, future Target contracts
mean money flows back to the contractor — the ticker is an associated
indicator." Now the profiler index attaches three flavors of ticker per
contractor and renders them as colored pills:

  green DIRECT    contractor IS the public issuer (Target Corp → TGT)
  amber PARENT    contractor is a subsidiary of a public parent
                    (Turner Construction → HOC.DE via Hochtief AG)
  blue  ASSOCIATED contractor co-appears on permits with a public
                    entity (TORNOW, KYLE F → TGT, 3 shared permits with
                    TARGET CORPORATION)

The associated flavor is the correlation signal J described — it pulls
the ticker for whoever the contractor has been working *with*, not
just what they are themselves. Most contractors are private; the
associated link is how the moat shows up.

Server-side:
- entity.ts new export `lookupTickerLite(name)` — cheap in-memory
  resolver that does only the SEC tickers index lookup + curated
  KNOWN_PARENT_MAP check, no per-call SEC profile or Stooq fetch.
  ~10ms per name after the index is loaded once.
- /intelligence/profiler_index now runs a third Socrata pull
  (5K permit pairs in window) to build a co-occurrence map. For each
  contractor in the result, attaches:
    .tickers.direct[]      — name matches a public issuer
    .tickers.associated[]  — top 5 co-permit partners that resolve
                              to a ticker, with partner_name +
                              co_permits count + partner_via reason

Front-end:
- mcp-server/profiler.html — new .ticker-pill styles (3 colors per
  attribution kind), pills render under the contractor name in the
  table. Hover title gives the full reason path.

Verified end-to-end on the public URL:
  search="tornow" → blue TGT pill, hint "Associated via co-permits
                    with TARGET CORPORATION (3 shared permits) —
                    TARGET CORP"
  search="target" → green TGT × 2 (TARGET CORPORATION +
                    CORPORATION TARGET name variants both resolve
                    direct to the same issuer)
  default top 200 → 15 ticker pills surface across the page including
                    JPM (via JPMORGAN CHASE BANK co-permits) and
                    parent-link tickers for the construction majors.
 2026-04-28 06:01:04 -05:00
root f6a7621b2d demo: profiler index — directory of every Chicago contractor 
J asked for "a profiler index that shows a history of everyone." This
is a /profiler directory page (also reachable via /contractors) that
ranks every contractor who's filed a Chicago permit, by total permit
value. Rows are clickable into the full /contractor profile.

Defaults: since 2025-06-01, min permit cost $250K, top 200 contractors
by total_cost. Server pulls two Socrata GROUP BY queries (one keyed on
contact_1_name, one on contact_2_name), merges them so contractors
listed in either applicant or contractor slot appear once with combined
counts/cost. ~300ms cold.

UI: live search box, since-date selector, min-cost selector, sortable
columns (name / permits / total_cost / last_filed). Live numbers as of
this write: 200 contractors, 1,702 permits, $14.22B aggregate. Filter
"Target" returns TARGET CORPORATION + CORPORATION TARGET (name variants
from Socrata).

Also fixes J's other complaint — "no new contracts, Target is gone":

  /intelligence/permit_contracts was hard-capped at $limit=6 + only
  the most recent 6 over $250K, so any day with 6 fresh permits would
  push older contractors (Target) off the panel entirely. Now defaults
  to 24 (caller can pass body.limit up to 100), so 2-3 days of permits
  stay on the panel. Added body.contractor — passes a name into the
  WHERE so the staffer can pin a specific contractor to the panel
  ("Target Corporation" → 3 of their permits over $250K).

Server-side:
- new POST /intelligence/profiler_index — paginated contractor index
  (since, min_cost, search, limit) with merged contact_1+contact_2
  aggregations
- /intelligence/permit_contracts — body.limit + body.contractor
- /profiler and /contractors routes serve profiler.html

Front-end:
- new mcp-server/profiler.html — sortable table, live filter, deep
  links to /contractor?name=... (prefix-aware via P, so /lakehouse
  works on devop.live)
- search.html + console.html nav: added "Profiler" link

Verified end-to-end via playwright on the public URL.
 2026-04-28 06:01:04 -05:00
root 31d8ef918c demo: contractor links — respect the /lakehouse path prefix 
J reported https://devop.live/contractor?name=3115%20W%20POLK%20ST.%20LLC
returned 404. Cause: the anchor href was a bare /contractor, which on
devop.live routes to the LLM Team UI (port 5000) at the main site root,
not the lakehouse mcp-server (which lives under /lakehouse/*).

Every page that renders a contractor link now uses the same prefix
detector the dashboard already had:

  var P = location.pathname.indexOf('/lakehouse') >= 0 ? '/lakehouse' : '';

Files updated:
- search.html: entity-brief anchor + preview anchor → P+/contractor
- console.html: permit-card contractor list → P+/contractor
- contractor.html: history.replaceState + back-link + the
  /intelligence/contractor_profile fetch all use P prefix. The page
  is reachable at /lakehouse/contractor on the public URL and bare
  /contractor on localhost; both work without further config.

Verified:
  https://devop.live/lakehouse/contractor?name=3115%20W%20POLK%20ST.%20LLC
    → 200, 29.9 KB, full profile renders. Contractor has 1 permit on
    file (a small LLC), 1 geocoded so the heat map plots one marker.
 2026-04-28 06:01:04 -05:00
root a1066db87b demo: contractor profile — heat map, project index, 12 awaiting sources 
The contractor.html click-target J asked for: a separate page (not a
modal, not a fall-through search) showing every angle on a contractor.
Reachable from the Co-Pilot dashboard, the staffers console, and the
search box — all anchor-wrap contractor names to /contractor?name=...

What's new on the page:

1. PROJECT INDEX — build-signal score
   Single 0-100 number with the drivers laid out beneath. Driver list
   is staffer-readable: "59 Chicago permits in 180d (+30) · OSHA 20
   inspections (-25) · federal contractor (+15)". Score weights are
   placeholders to be replaced by an ML model once the 12 awaiting
   sources ship — the current 6 wired signals would not give a real
   model enough features.

2. HEAT MAP — every Chicago permit they've been contact_1 or contact_2
   on, last 24 months, plotted on a leaflet dark map. Color by cost
   (green <$100K, amber $100K-$1M, red ≥$1M), radius proportional to
   cost so the staffer sees where money + activity concentrates. Click
   a marker for permit detail (cost, date, work type, address, permit
   ID). All 50 of Turner Construction's geocoded recent permits in
   Chicago plot end-to-end.

3. ACTIVITY TIMELINE — monthly permit count, bar chart, with the
   first/last month labels so the staffer sees momentum. Tooltip on
   each bar gives the count and total cost for that month.

4. 12 AWAITING SOURCES — placeholder cards for the public datasets
   that would 3× the build-signal feature count. Each card has:
     - source name (real, e.g. DOL Wage & Hour, EPA ECHO, MSHA, BBB)
     - one-liner in coordinator language ("Has this contractor stiffed
       workers? Will they pay our staffing invoices?")
     - "Would show:" sample shape so the engineering scope is concrete
   Order is staffing-decision relevance:
     1. DOL Wage & Hour (WHD violations)
     2. State Licensure Boards (active license + expiry)
     3. Surety Bond Capacity (bonding ceiling)
     4. EPA ECHO Compliance (env violations at sites)
     5. DOT/FMCSA Carrier Safety (crash + OOS rates)
     6. BBB Complaints + Rating
     7. PACER Civil Suits (FLSA / Title VII / ADA)
     8. UCC Lien Filings (cash flow distress)
     9. D&B / Credit Bureau (PAYDEX, payment behavior)
    10. State UI Employer Claims (workforce stability)
    11. MSHA Mine Safety (excavation / aggregate / heavy)
    12. Registered Apprenticeships (DOL RAPIDS pipeline)

Server-side: entity.ts fetchContractorHistory now pulls the 50 most
recent permits with id + lat/lng + work_description, so the heat map
and timeline have what they need without a second SQL hop. The
ContractorHistory.recent_permits type gained the optional fields.

Front-end: contractor.html got 4 new render sections, leaflet wiring
(stylesheet + script in head), placeholder grid CSS, and a PLACEHOLDERS
const at the bottom with the 12 sources. All popup HTML is built via
DOM construction (textContent + appendChild) — no innerHTML, no XSS.

console.html: contractor names from /intelligence/permit_contracts now
anchor-wrapped to /contractor?name=... so the click-through J described
works from the staffers console too. Click stops propagation so the
permit details element doesn't toggle on the same click.

Verified end-to-end via playwright — Turner Construction profile shows:
  PIX score "Mixed signals — review drivers below"
  Heat map: "50 permits plotted · green/amber/red"
  4 section labels in order
  12 placeholder cards in the documented order
 2026-04-28 06:01:04 -05:00
root 5f0beffe80 demo: G — per-staffer hot-swap index (synthetic coordinator personas) 
Same corpus, different relevance gradient per staffer. Three personas
defined in mcp-server/index.ts STAFFERS roster (Maria/IL, Devon/IN,
Aisha/WI), each with a primary state + secondary cities. Server-side:
/intelligence/chat smart_search accepts a staffer_id body field; when
set, defaults state to the staffer's territory and labels the playbook
context as theirs. The playbook patterns query also defaults its geo
to the staffer's primary city/state, so the recurring-skills/cert
breakdowns reflect what they actually fill, not the global IL prior.

Front-end: a staffer selector dropdown beside the existing state/role
filters. Picking a staffer auto-pins state to their territory, shows
a greeting line, relabels the MEMORY panel as MARIA'S/DEVON'S/AISHA'S
MEMORY, and sends staffer_id to chat for scoping.

Dropdown is populated from /staffers (NOT /api/staffers — the generic
/api/* passthrough sends everything under /api/ to the Rust gateway,
which doesn't own the roster). loadStaffers runs at window-load
independently of loadDay's Promise.all so the dropdown populates even
if simulation/SQL inits error out.

Verified end-to-end via playwright. Same q="forklift operators":
  no staffer  → 509 workers across MI/OH/IA, MEMORY label
  as Devon    → 89 IN-only (Fort Wayne, Terre Haute), DEVON'S MEMORY
  as Aisha    → 16 WI-only (Milwaukee, Madison, Green Bay), AISHA'S MEMORY
As Maria with q="8 production workers near 60607":
  tags: headcount: 8 · zip 60607 → Chicago, IL · role: production · city: Chicago
  20 workers, MARIA'S MEMORY label, top results in Chicago zips

Closes the demo-side build of A-G from the persona plan:
  A. zip → city/state, B. headcount, C. bare-name, D. temporal,
  E. late-worker triage, F. contractor anchor, G. per-staffer index.
 2026-04-28 06:01:04 -05:00
root 677065de76 demo: P2 — staffer-language routes (zip, headcount, name, late-triage, ingest log) 
Built from a playwright run as three personas:
  Maria   — "8 production workers near 60607 by next Friday, prior-fill at this client"
  Devon   — "what came in last night?"
  Aisha   — "Marcus running late site 4422"

Each one previously fell through to smart_search and returned irrelevant
results (geo wrong, headcount ignored, no triage, no temporal). Now:

A. Zip code → city/state lookup. Chicago zips (606xx, 607xx, 608xx)
   resolve to {city: Chicago, state: IL}; 13 metro prefixes covered.
   Maria's "near 60607" now returns Chicago workers, not Dayton/Green Bay.

B. Headcount parser. "8 production workers" / "12 forklift operators" /
   "5 welders" set top_k 1..200, capped 5..25 for SQL+vector LIMIT.
   Allows 0-2 role words between the count and the worker noun so
   "8 production workers" matches as well as "8 workers".

C. Bare-name profile lookup. Single short capitalized phrase
   ("Marcus" / "Sarah Lopez") triggers a profile route. Per-token LIKE
   AND-joined so "Marcus Rivera" matches "Marcus L. Rivera" without
   hardcoding middle initials.

E. Late-worker / no-show triage. Pattern: <Name> (running late|late|
   no show|sick|out today|called out|can't make it) — pulls profile +
   reliability + responsiveness + recent calls, sources 5 same-role
   same-geo backfills sorted by responsiveness, drafts a client SMS
   the coordinator can copy. Front-end renders triage card + Copy SMS
   button + green backfill list.

F. Contractor name preview anchor. The PROJECT INDEX preview line on
   each permit card now wraps contact_1_name and contact_2_name in
   anchors to /contractor?name=... — clicking a contractor finally
   navigates instead of doing nothing. Click handler stops propagation
   so the details element doesn't toggle.

D. Temporal "what came in" route. last night / today / past N hours /
   recent — surfaces datasets from the catalog whose updated_at is
   within the window, samples one row per dataset to detect worker-
   shape, groups by role for worker tables. Schema-agnostic — drop
   any dataset and it shows up. Currently sparse because no fresh
   ingest has happened today; will populate as ingest runs.

Server: /intelligence/chat smart_search route accepts structured
state/role from the search-form dropdowns (P1 from prior commit) and
now ALSO honors b.state, b.role, q.match for headcount + zip + name +
triage patterns BEFORE falling through to NL parsing.

Front-end: doSearch dispatches on response.type and renders triage,
profile, ingest_log, and miss states with type-specific UI. All DOM
construction uses textContent / appendChild — no innerHTML, no XSS.

Verified end-to-end via playwright drive of devop.live/lakehouse:
  Maria  → 8 Chicago Production Workers (60685, 60662, 60634)
           tags: "headcount: 8 · zip 60607 → Chicago, IL · ..."
  Aisha  → Marcus V. Campbell card + draft SMS + 5 Quincy IL backfills
           "I'm dispatching Scott B. Cooper (96% reliability) to cover."
  Devon  → ingest_log surfaces successful_playbooks_live (last 1h)
  Marcus → 5 profiles (Adams Louisville KY, Jenkins Green Bay WI, ...)

Screenshots: /tmp/persona_v2/{01_maria,02_aisha,03_devon,04_marcus}.png

Restart sequence after these edits: pkill -9 -f "mcp-server/index.ts" ;
cd /home/profit/lakehouse ; bun run mcp-server/index.ts. The bun on
:3700 is not systemd-managed (pre-existing convention).
 2026-04-28 06:01:04 -05:00
root fb99e92a60 demo: P1 — search filter now actually filters by state and role 
The Co-Pilot search box read state and role from the dropdowns (#sst, #srl)
but appended them to the message string as ' in '+st. The server's NL
parser then matched the literal preposition "in" against the case-insensitive
regex /\b(IL|IN|...)\b/i and assigned state IN (Indiana) to every search.
Result: typing "forklift in IL" returned Indiana workers. Same for WI, TX,
any state — all silently became Indiana. That was the "cached/generic
response" the legacy staffing client was seeing.

Two prongs:

1. search.html doSearch() now passes structured fields:
     {message, state, role}
   instead of munging into the message text. Dropdown selections bypass
   NL parsing entirely.

2. /intelligence/chat smart_search route accepts those structured fields
   and prefers them over regex archaeology. Falls back to NL parsing only
   when fields aren't provided. Fixed the regex too: the prepositional
   form (?:in|from)\s+(STATE) wins, the standalone form requires uppercase
   (drops /i flag) so the lowercase preposition "in" can no longer match.

Verified live:
- POST /intelligence/chat {"message":"forklift","state":"IL"}
    → 167 IL forklift operators (Galesburg, Joliet, ...)
- POST /intelligence/chat {"message":"forklift","state":"WI","role":"Forklift Operator"}
    → 16 WI Forklift Operators (Milwaukee, Madison, ...)
- POST /intelligence/chat {"message":"forklift in IL"} (NL fallback)
    → 167 IL workers (regex now correctly distinguishes preposition from state code)

Playwright drove the live UI through devop.live/lakehouse and confirmed the
front-end posts the structured body and the result panel renders the right
state. Restart sequence: kill old bun :3700, bun run mcp-server/index.ts.
 2026-04-28 06:01:04 -05:00
profit ed57eda1d8 Merge PR #11: distillation v1.0.0 + Phase 42-45 + auditor cross-lineage + staffing cutover 
Closes the long-running scrum/auto-apply-19814 branch.

118 commits including:
- Distillation v1.0.0 substrate (tag distillation-v1.0.0 / e7636f2) — 145 tests, 22/22 acceptance, 16/16 audit-full
- Auditor rebuild on substrate (88s vs 25min, 50x fewer cloud calls)
- Phase 42-45 closure (validator crate + /v1/validate + /v1/iterate + /v1/health + /doc_drift/scan + Phase 44 /v1/chat migration)
- Auditor cross-lineage fabric (Kimi K2.6 / Haiku 4.5 / Opus 4.7 auto-promotion + per-PR cap with auto-reset on push)
- 5-provider routing (added opencode + kimi-direct adapters)
- Mode runner with composed-corpus downgrade gate (codereview_isolation default; composed lost 5/5 on grok-4.1-fast)
- Staffing cutover decisions A/C/D + B safe views — workers_500k_v9 corpus rebuild deferred to background job

Verified before merge:
- audit-full 16/16 required pass
- cargo check -p validator -p gateway clean
- All kimi_architect BLOCK findings dismissed as confabulation, logged in data/_kb/human_overrides.jsonl
- Kimi forensic HOLD on v1.0.0 verified manually: 2/8 false + 6/8 latent guarantees that do not fire under prod data
 2026-04-27 15:55:22 +00:00
26 changed files with 4185 additions and 477 deletions
Expand all files Collapse all files
.gitignore
+5 -1
View File
@@ -6,5 +6,9 @@ __pycache__/
*.pyc *.pyc
# Headshot pool — binary face JPGs are fetched by scripts/staffing/fetch_face_pool.py # Headshot pool — binary face JPGs are fetched by scripts/staffing/fetch_face_pool.py
# (synthetic StyleGAN, ~100MB for 200 faces). Manifest + fetch script are tracked. # (synthetic StyleGAN, ~580MB for 1000 faces). Manifest + fetch script are tracked.
data/headshots/face_*.jpg data/headshots/face_*.jpg
data/headshots/_thumbs/
# ComfyUI on-demand generated portraits (per-worker unique). Cached on first
# request; fully regeneratable via /headshots/generate/:key.
data/headshots_gen/
Cargo.lock
Generated
+1
View File
@@ -48,6 +48,7 @@ version = "0.1.0"
dependencies = [ dependencies = [
"async-trait", "async-trait",
"axum", "axum",
"lru",
"reqwest", "reqwest",
"serde", "serde",
"serde_json", "serde_json",
STATE_OF_PLAY.md
+239
View File
@@ -0,0 +1,239 @@
# STATE OF PLAY — Lakehouse
**Last verified:** 2026-04-27 ~20:35 CDT
**Verified by:** live probe, not memory.
> **Read this FIRST.** When the user says "we're working on lakehouse," they mean the working code captured below — NOT what `git log` framed as "the cutover" or what memory snapshots from 2 days ago suggest. If memory contradicts this file, this file wins. Update it when something is verified working — not when a phase finishes.
---
## VERIFIED WORKING RIGHT NOW
### The client demo (Staffing Co-Pilot)
**Public URL:** `https://devop.live/lakehouse/` — 200, "Staffing Co-Pilot" (159 KB SPA, leaflet maps, dark theme).
**Local URL:** `http://localhost:3700/` — same page, served by `mcp-server/index.ts` (PID 1271, started 09:48 CDT today).
**The staffers console** (the one the client was thoroughly impressed with):
- `https://devop.live/lakehouse/console` — 200, "Lakehouse — What Your Staffing System Would Do" (26 KB)
- Pulls project index via `/api/catalog/datasets` (36 datasets) + playbook memory via `/api/vectors/playbook_memory/stats` (4,701 entries with embeddings, real ops like *"fill: Maintenance Tech x2 in Milwaukee, WI"*)
Client-visible flow that works end-to-end on the public URL:
| Endpoint | Sample output |
|---|---|
| `GET /api/catalog/datasets` | 36 datasets indexed: timesheets 1M, call_log 800K, workers_500k 500K, email_log 500K, workers_100k 100K, candidates 100K, placements 50K, job_orders 15K, successful_playbooks_live 2,077 |
| `GET /api/vectors/playbook_memory/stats` | 4,701 fill operations with embeddings |
| `GET /system/summary` | 36 datasets, 2.98M rows, 60 indexes, 500K workers loaded, 1K candidates |
| `POST /intelligence/staffing_forecast` | 744 Production Workers needed in 30d, 11,281 bench (4,687 reliable), coverage 1,444%, risk=ok. Same for Electrician (need 32, bench 2,440) and Maintenance Tech (need 17, bench 5,004). |
| `POST /intelligence/permit_contracts` | permit `3442956` $500K → 3 Production Workers, 886-candidate pool, 95% fill, $36K gross. 5 more Chicago permits with 8 workers each, same pool, 95% fill, $96K each. |
| `POST /intelligence/market` | major Chicago permits ranked: $730M O'Hare, $615M 307 N Michigan, $580M casino, $445M Loop transit (real geo coords). |
| `POST /intelligence/permit_entities` | architects + contractors from permit contacts (e.g. "KACPRZYNSKI, ANDY", "SLS ELECTRICAL SERVICE"). |
| `POST /intelligence/activity` + `/intelligence/arch_signals` + `/intelligence/chat` | all 200 |
The demo tells the story: *"upcoming Chicago contracts → workers needed → coverage from the bench → architects/contractors involved → revenue and margin."* That's the "live data + anticipating contracts + complete workflow" pitch — working as of right now.
### Backend, verified live this session
| Surface | State |
|---|---|
| Gateway `:3100` | up, 4 providers configured, `/v1/health` 200 with 500K workers loaded |
| MCP server `:3700` (Co-Pilot demo) | up, all `/intelligence/*` endpoints respond |
| VCP UI `:3950` | started this session, `/data/*` 200, real numbers |
| Observer `:3800` | ring full (2,000/2,000) — older events evicted, query Langfuse for 24h-ago state |
| Sidecar `:3200` | up |
| Langfuse `:3001` | recording, `gw:/log` + `v1.chat:openrouter` traces visible |
| LLM Team UI `:5000` | up, only `extract` mode registered |
| OpenCode fleet | **40 models reachable through one `sk-*` key** (verified live `GET https://opencode.ai/zen/v1/models`) |
OpenCode catalog (live):
- Claude: opus-4-7, opus-4-6, opus-4-5, opus-4-1, sonnet-4-6, sonnet-4-5, sonnet-4, haiku-4-5
- GPT-5: 5.5-pro, 5.5, 5.4-pro, 5.4, 5.4-mini, 5.4-nano, 5.3-codex-spark, 5.3-codex, 5.2, 5.2-codex, 5.1-codex-max, 5.1-codex, 5.1-codex-mini, 5.1, 5-codex, 5-nano, 5
- Gemini: 3.1-pro, 3-flash
- GLM: 5.1, 5
- Minimax: m2.7, m2.5
- Kimi: k2.6, k2.5
- Qwen: 3.6-plus, 3.5-plus
- Other: BIG-PKL (was a typo-prone name in the catalog, model id starts with "big-pkl-something")
- Free tier: minimax-m2.5-free, hy3-preview-free, ling-2.6-flash-free, trinity-large-preview-free
### The substrate (frozen — do not re-architect)
- Distillation v1.0.0 at tag `e7636f2` — **145/145 bun tests pass, 22/22 acceptance, 16/16 audit-full**
- Output: `data/_kb/distilled_{facts,procedures,config_hints}.jsonl` + `data/vectors/distilled_{factual,procedural,config_hint}_v20260423102847.parquet`
- Auditor cross-lineage: Kimi K2.6 ↔ Haiku 4.5 alternation, Opus auto-promote on diffs >100k chars, **per-PR cap=3 with auto-reset on new head SHA**
- Pathway memory: 88 traces, 11/11 successful replays (probation gate crossed)
- Mode runner: 5 native modes; `codereview_isolation` is default; composed-corpus auto-downgrade verified Apr 26 (composed lost 5/5 vs isolation, p=0.031)
### Matrix indexer
30+ live corpora including:
- 5 versions of `workers_500k_v1..v9` (50K embedded chunks each)
- 11 batched 2K-row shards `w500k_b3..b17`
- `chicago_permits_v1` (3,420), `resumes_100k_v2` (100K candidates), `ethereal_workers_v1` (10K)
- `lakehouse_arch_v1` (2,119), `lakehouse_symbols_v1` (2,470), `lakehouse_answers_v1` (1,269), `scrum_findings_v1` (1,260)
- `kb_team_runs_v1` (12,693) + `kb_team_runs_agent` (4,407) — LLM-team play history embedded
- `distilled_factual_v20260423102507` (8) — distillation output
### Code health
- `cargo check --workspace` → **0 warnings, 0 errors**
- `bun test auditor + tests/distillation` → **145/145 pass**
- `ui/server.ts` + `auditor.ts` bundle clean
---
## DO NOT RELITIGATE
- **PR #11 is merged into `origin/main` as `ed57eda`** — do not "still need to merge PR #11."
- **Distillation tag `distillation-v1.0.0` at `e7636f2` is FROZEN** — do not re-architect schemas, scorer rules, audit fixtures.
- **Kimi forensic HOLD verdict (2026-04-27) was 2/8 false + 6/8 latent** — do not re-debate, see `reports/kimi/audit-last-week-full.md`.
- **`candidates_safe` `vertical` column bug** — fixed at catalog metadata layer in commit `c3c9c21`. Do not "discover" it again.
- **Decisions A/B/C/D from `synthetic-data-gap-report.md`** — all four scripts shipped today (`d56f08e`, `940737d`, `c3c9c21`). Do not "ask J for approval."
- **`workers_500k.phone` type fixup** — already string. The fixup script is idempotent; running it is a no-op.
- **`client_workerskjkk` typo dataset** — was breaking every SQL query (catalog had it registered, file didn't exist). Removed via `DELETE /catalog/datasets/by-name/client_workerskjkk` this session. Do not re-add. Adding a startup gate that errors on unrecognized parquet names is the long-term fix per now.md Step 2C.
---
## FIXES MADE THIS SESSION (2026-04-27 evening)
1. **`crates/gateway/src/v1/iterate.rs:93`** — `state` → `_state` (cleared the one cargo warning).
2. **`lakehouse-ui.service` (Dioxus)** — disabled. Was failing 7,242 times against a missing `target/dx/ui/debug/web/public` build dir. `systemctl stop && disable`.
3. **VCP UI on `:3950`** — started `bun run ui/server.ts` (PID 1162212, log `/tmp/lakehouse_ui.log`). `/data/*` endpoints now 200 with real data.
4. **`client_workerskjkk` catalog entry** — `DELETE /catalog/datasets/by-name/client_workerskjkk` removed the dead manifest. **This was the actual root cause** of `/system/summary` reporting `workers_500k_rows: 0` and the demo showing zero bench. Every SQL query was failing schema inference on the missing file before reaching its target table. Fixed → `workers_500k_rows: 500000`, `candidates_rows: 1000`, demo coverage flipped from "critical 0%" to actual percentages on devop.live/lakehouse.
## FIXES MADE THIS SESSION (2026-04-28 early — face pool)
5. **Synthetic StyleGAN face pool — 1000 faces, gender+race+age tagged.** `scripts/staffing/fetch_face_pool.py` fetches from thispersondoesnotexist.com; `scripts/staffing/tag_face_pool.py --min-age 22` runs deepface and excludes minors. `data/headshots/manifest.jsonl` now has gender (494 men / 458 women), race (caucasian 662 · east_asian 128 · hispanic 86 · middle_eastern 59 · black 14 · south_asian 3), age, and 48 minor exclusions. Server pool = 952 servable faces.
6. **`mcp-server/index.ts:1308` `/headshots/:key` route** — gender×race×age intersection bucketing with graceful fallback (gender-only → all). Same key always returns same face; different keys spread evenly.
7. **`/headshots/_thumbs/` pre-resized 384×384 webp** (60× smaller: 587KB → ~11KB). Without this, 40-card grids overran Chrome's parallel-connection budget and ~75% of tiles never finished decoding. Generated via parallel ffmpeg (`xargs -P 8`); `.gitignore`d.
8. **`mcp-server/search.html` + `console.html`** — dropped `img.loading='lazy'`. With 11KB thumbs, eager load is cheap (~500KB for 50 cards) and avoids the off-screen race that lazy decode produced.
9. **ComfyUI on-demand uniqueness — `serve_imagegen.py:32`** added `seed` to `_cache_key()` (was caching by prompt only — 3 different worker seeds collapsed to 1 cached image). Verified: seed=839185194/195/196 → 3 distinct md5s.
10. **`mcp-server/index.ts:1234` `/headshots/generate/:key`** — ComfyUI hot-path that derives a deterministic-per-worker seed via djb2-style hash; cold ~1.5s, cached ~1ms. Worker prompt format: `professional corporate headshot portrait of a {age}-year-old {race} {gender}, {role}, neutral expression, plain studio background, soft natural lighting, sharp focus, photorealistic, dslr`. Cache at `data/headshots_gen/` (gitignored, regeneratable).
11. **Confidence-default name resolution** in `search.html` — `genderFor()` and `guessEthnicityFromFirstName()` lookup tables (FEMALE_NAMES, MALE_NAMES, NAMES_HISPANIC, NAMES_BLACK, NAMES_SOUTH_ASIAN, NAMES_EAST_ASIAN, NAMES_MIDDLE_EASTERN). Xavier → man+hispanic, Aisha → woman+black, etc. Every worker resolves to a face-pool bucket.
End-to-end verified: playwright run on `https://devop.live/lakehouse/?q=forklift+operators+IL` → 21/21 cards loaded, 0 broken, all 384×384 webp thumbs.
---
## OPEN — but not blocking the demo
| Item | What | When to act |
|---|---|---|
| `modes.toml` `staffing_inference.matrix_corpus` | still says `workers_500k_v8`. v9 in vector index is from Apr 17 (raw-sourced, not safe-view). The new `build_workers_v9.sh` rebuilds from `workers_safe`. | Run when you have 30+ min for the rebuild. |
| Open PRs #6, #7, #10 | sitting since Apr 22-24, auditor verdicts on disk at `data/_auditor/kimi_verdicts/{6,7,10}-*.json` | Read verdicts, decide reconcile/close. |
| `test/enrich-prd-pipeline` branch | 35 unmerged commits, includes more-evolved auditor/inference.ts (666 vs main's 580 lines), curation+fact-extractor wiring | Reconcile or formally archive — see `memory/project_unmerged_architecture_work.md`. |
| `federation-hnsw-trials` stash | Lance + S3/MinIO prototype, `aws-config` crate added, 708 insertions | Phase B from EXECUTION_PLAN.md — revisit when Parquet vector ceiling actually hurts. |
| `candidates` manifest drift | manifest 100K vs SQL 1K. Cosmetic. | Run a metadata resync if it matters. |
---
## RUNTIME CHEATSHEET
```bash
# Verify the demo (public + local both work)
curl -sS https://devop.live/lakehouse/ # Co-Pilot HTML
curl -sS https://devop.live/lakehouse/console # staffers console
curl -sS -X POST https://devop.live/lakehouse/intelligence/staffing_forecast \
-d '{}' -H 'content-type: application/json' \
| jq '.forecast[] | {role, demand_workers, bench_total, coverage_pct, risk}'
# Restart sequence (after Rust changes)
sudo systemctl restart lakehouse.service # gateway :3100
sudo systemctl restart lakehouse-auditor # auditor daemon
sudo systemctl restart lakehouse-observer # observer :3800
# UI bun on :3950 is NOT systemd-managed (lakehouse-ui.service is disabled).
# Restart manually: kill <pid>; nohup bun run ui/server.ts > /tmp/lakehouse_ui.log 2>&1 &
# Health checks
curl -sS http://localhost:3100/v1/health | jq # workers_count, providers
curl -sS http://localhost:3100/vectors/pathway/stats | jq
curl -sS http://localhost:3100/v1/usage | jq # since-restart cost
curl -sS http://localhost:3700/system/summary | jq # dataset counts
```
---
## VISION — what we're actually building (not what's done)
J's framing for the legacy staffing company:
- Pull live data, anticipate contracts based on Chicago permits → real architect/contractor associations, headcount, time period, money, scope.
- Hybrid + memory index → search large corpora cheaply.
- Email comes in → verify against contract; SMS comes in → alert when index changes.
- Real-time.
- Invent metrics nobody else has using the hybrid index.
- Next stage: workers download an app → geolocation clock-in → automatic responsiveness measurement, no user effort, with incentives for using it.
- Find people getting certificates (passive cert tracking).
- Pull union data → bring contracts that work for **employees**, not just employers.
- All metrics visible, nothing hidden, value-aligned with what each side actually needs.
If a future session is shaving away from this vision toward "fix the cutover" or "land Phase X," the vision wins. Phases are scaffolding for the vision, not the goal.
---
## CURRENT PLAN — fix the demo for the legacy staffing client
Built from playwright audit of the live demo (2026-04-27 evening). Each item ends in something the client can SEE, not internal cleanups.
**Demo state is anchored by git tag `demo-2026-04-27`** (commit `ed57eda`, the merge of PR #11). To restore code state: `git checkout demo-2026-04-27`. To restore runtime state: `DELETE /catalog/datasets/by-name/client_workerskjkk` (catalog hot-fix is not in git).
### P1 — Search box that actually filters (highest visible impact)
**Problem:** typing in `#sq` and pressing Enter fires `POST /intelligence/chat` with body `{"message":"<query>"}`. The state (`#sst`) and role (`#srl`) selects are ignored — never sent in the body. So every search returns a generic chat completion, never a SQL+vector hybrid filter against `workers_500k`. That is the "cached/generic response" the client sees.
**Fix:** in `mcp-server/search.html`, change the search-submit handler to call the real worker search endpoint with `{query, state, role, top_k}`. The MCP `search_workers` tool surface already exists; route the form there. Render returned worker rows in the existing card grid.
**Done when:** typing "forklift" + state IL + role "Forklift Operator" returns ≤ top_k IL Forklift Operators, and changing state to WI returns different workers.
### P2 — Contractor-name click → `/contractor` profile page
**Problem:** clicking a contractor name in any rendered card stays on `/lakehouse/`. URL doesn't change.
**Fix:** wrap contractor names in `<a href="/contractor?name=<encoded>">`. The page `mcp-server/contractor.html` (14.8 KB, "Contractor Profile · Staffing Co-Pilot") already exists at `/contractor` and the data endpoint `/intelligence/contractor_profile` already returns rich data.
**Then check contractor.html actually shows:** full history of every record the database has on that contractor + heat map of locations underneath + relevant info (per J 2026-04-27). If the page is incomplete, finish it. Otherwise just wire the link.
**Done when:** clicking "KACPRZYNSKI, ANDY" opens a profile with: every Chicago permit they're contact_1 or contact_2 on, a leaflet map with markers for each address, and any matched workers from prior placements at their sites.
### P3 — Substrate signal at the bottom shows the right numbers
**Problem:** J reports the bottom panel says "playbook memory empty, 80 traces 0 replies." Reality from the live endpoints: `/api/vectors/playbook_memory/stats` = 4,701 entries with embeddings; `/vectors/pathway/stats` = 88 traces, 11/11 replays.
**Fix:** find the renderer in search.html that builds the substrate signal panel; verify it's hitting the right endpoints and reading the right keys; fix shape mismatches.
**Done when:** bottom panel shows real numbers (4,701 playbooks, 88 traces, 11/11 replays) and references at least one specific recent operation from the playbook stats sample.
### P4 — Top nav reflects today's architecture
**Problem:** Walkthrough/Architecture/Spec/Onboard/Alerts/Workspaces tabs all return 200 but content is from old architecture. Doesn't mention: gateway scratchpad, memory indexer, ranker, mode runner, OpenCode 40-model fleet, distillation substrate, auditor cross-lineage.
**Fix:** rewrite `mcp-server/proof.html` (or add a single new page "What's running" that replaces Architecture+Spec) to describe what's actually shipped as of `demo-2026-04-27`. Keep one architecture page, drop redundancy. Either complete or hide Onboard/Alerts/Workspaces — J's call which.
**Done when:** the architecture page tells a non-technical reader, in 2 minutes, what each piece does in coordinator-relatable terms ("intern that read every email", not "3-stage adversarial inference pipeline").
### P5 — Caching for the project-index build_signal (J flagged unfinished)
**Problem:** "we never finished our caching for project index build signal it's not pulling new information." Need to find what `build_signal` refers to. Likely a scrum/auditor signal that should rebuild the `lakehouse_arch_v1` corpus on commit but isn't wired to.
**Fix:** identify the build-signal pipeline (likely in `auditor/` or `crates/vectord/`), wire its emit to a corpus rebuild, verify by making a test commit and watching the new chunk appear in `/vectors/indexes` for `lakehouse_arch_v1`.
**Done when:** committing a new file to `crates/` causes `lakehouse_arch_v1` chunk_count to increase within N minutes.
### P0 — Anchor the demo state (DONE)
Tagged `ed57eda` as `demo-2026-04-27`. Future sessions: `git checkout demo-2026-04-27` to land in this exact code state.
---
## EXECUTION ORDER
1. **P1 first** — biggest visible bug, ~30-60 min
2. **P2 next** — contractor click is the second-biggest "doesn't work" the client sees, ~20 min if profile is mostly done
3. **P3** — small fix, big "looks alive" win
4. **P4** — biggest scope; might split across sessions
5. **P5** — feature work, only after the visible bugs are fixed
Each item commits independently with the format `demo: P<n> — <one-line>` so the commit log doubles as a progress journal. After each merge to main, re-tag `demo-latest` to point at the new HEAD.
Stop here and let J pick which item to start with. Do not silently extend scope.
bot/propose.ts
+5 -3
View File
@@ -16,12 +16,14 @@ import type { Gap, Proposal } from "./types.ts";
// Phase 44 migration (2026-04-27): bot/propose.ts now flows through // Phase 44 migration (2026-04-27): bot/propose.ts now flows through
// the gateway's /v1/chat instead of hitting the sidecar's /generate // the gateway's /v1/chat instead of hitting the sidecar's /generate
// directly. /v1/usage tracks the call, Langfuse traces it, observer // directly. /v1/usage tracks the call, Langfuse traces it, observer
// sees it. Same upstream model (CLOUD_MODEL gpt-oss:120b on // sees it. Gateway owns the routing.
// Ollama Cloud) — gateway just owns the routing. //
// 2026-04-28: gpt-oss:120b → deepseek-v3.2 via Ollama Pro. Newer
// DeepSeek revision, faster, still on the same OLLAMA_CLOUD_KEY.
const GATEWAY_URL = process.env.LH_GATEWAY_URL ?? "http://localhost:3100"; const GATEWAY_URL = process.env.LH_GATEWAY_URL ?? "http://localhost:3100";
const REPO_ROOT = "/home/profit/lakehouse"; const REPO_ROOT = "/home/profit/lakehouse";
const PRD_PATH = `${REPO_ROOT}/docs/PRD.md`; const PRD_PATH = `${REPO_ROOT}/docs/PRD.md`;
const CLOUD_MODEL = process.env.LH_BOT_MODEL ?? "gpt-oss:120b"; const CLOUD_MODEL = process.env.LH_BOT_MODEL ?? "deepseek-v3.2";
const MAX_TOKENS = 6000; const MAX_TOKENS = 6000;
export async function findGaps(): Promise<Gap[]> { export async function findGaps(): Promise<Gap[]> {
config/modes.toml
+7 -2
View File
@@ -44,7 +44,10 @@ name = "staffing_inference"
# pattern generalizes beyond code review. # pattern generalizes beyond code review.
preferred_mode = "staffing_inference_lakehouse" preferred_mode = "staffing_inference_lakehouse"
fallback_modes = ["ladder", "consensus", "pipeline"] fallback_modes = ["ladder", "consensus", "pipeline"]
default_model = "openai/gpt-oss-120b:free" # 2026-04-28: gpt-oss-120b:free → kimi-k2.6 via Ollama Pro. Coding-
# specialized, faster than gpt-oss, on the same OLLAMA_CLOUD_KEY so
# no extra provider hop.
default_model = "kimi-k2.6"
matrix_corpus = "workers_500k_v8" matrix_corpus = "workers_500k_v8"
[[task_class]] [[task_class]]
@@ -58,7 +61,9 @@ matrix_corpus = "kb_team_runs_v1"
name = "doc_drift_check" name = "doc_drift_check"
preferred_mode = "drift" preferred_mode = "drift"
fallback_modes = ["validator"] fallback_modes = ["validator"]
default_model = "gpt-oss:120b" # 2026-04-28: gpt-oss:120b → gemini-3-flash-preview via Ollama Pro.
# Speed leader on factual checking, same OLLAMA_CLOUD_KEY.
default_model = "gemini-3-flash-preview"
matrix_corpus = "distilled_factual_v20260423095819" matrix_corpus = "distilled_factual_v20260423095819"
[[task_class]] [[task_class]]
config/providers.toml
+10 -5
View File
@@ -27,10 +27,15 @@ name = "ollama_cloud"
base_url = "https://ollama.com" base_url = "https://ollama.com"
auth = "bearer" auth = "bearer"
auth_env = "OLLAMA_CLOUD_KEY" auth_env = "OLLAMA_CLOUD_KEY"
default_model = "gpt-oss:120b" default_model = "deepseek-v3.2"
# Cloud-tier Ollama. Key resolved from OLLAMA_CLOUD_KEY env at gateway # Cloud-tier Ollama (Pro plan as of 2026-04-28). Key resolved from
# boot. Model-prefix routing: "cloud/<model>" auto-routes here # OLLAMA_CLOUD_KEY at gateway boot; Pro tier upgraded the account so
# (see gateway::v1::resolve_provider). # rate limits + model access widen without a key change. Model-prefix
# routing: "cloud/<model>" auto-routes here. 39-model fleet now
# includes deepseek-v3.2, deepseek-v4-{flash,pro}, gemini-3-flash-
# preview, glm-{5,5.1}, kimi-k2.6, qwen3-coder-next.
# 2026-04-28: default upgraded gpt-oss:120b → deepseek-v3.2 (newest
# DeepSeek revision; kimi-k2:1t still upstream-broken with HTTP 500).
[[provider]] [[provider]]
name = "openrouter" name = "openrouter"
@@ -38,7 +43,7 @@ base_url = "https://openrouter.ai/api/v1"
auth = "bearer" auth = "bearer"
auth_env = "OPENROUTER_API_KEY" auth_env = "OPENROUTER_API_KEY"
auth_fallback_files = ["/home/profit/.env", "/root/llm_team_config.json"] auth_fallback_files = ["/home/profit/.env", "/root/llm_team_config.json"]
default_model = "openai/gpt-oss-120b:free" default_model = "x-ai/grok-4.1-fast"
# Multi-provider gateway. Covers Anthropic, Google, OpenAI, MiniMax, # Multi-provider gateway. Covers Anthropic, Google, OpenAI, MiniMax,
# Qwen, Gemma, etc. Key resolved via crates/gateway/src/v1/openrouter.rs # Qwen, Gemma, etc. Key resolved via crates/gateway/src/v1/openrouter.rs
# resolve_openrouter_key() — env first, then fallback files. # resolve_openrouter_key() — env first, then fallback files.
crates/aibridge/Cargo.toml
+1
View File
@@ -12,3 +12,4 @@ serde_json = { workspace = true }
tracing = { workspace = true } tracing = { workspace = true }
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] } reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
async-trait = "0.1" async-trait = "0.1"
lru = "0.12"
crates/aibridge/src/client.rs
+163 -1
View File
@@ -1,5 +1,10 @@
use lru::LruCache;
use reqwest::Client; use reqwest::Client;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use std::num::NonZeroUsize;
use std::sync::Mutex;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
use std::time::Duration; use std::time::Duration;
/// HTTP client for the Python AI sidecar. /// HTTP client for the Python AI sidecar.
@@ -18,11 +23,42 @@ use std::time::Duration;
/// callers that want observability (vectord modules); it's left /// callers that want observability (vectord modules); it's left
/// unset by callers that ARE the gateway internals (avoids self-loops /// unset by callers that ARE the gateway internals (avoids self-loops
/// + redundant hops). /// + redundant hops).
/// Per-text embed cache key. We key on (model, text) so different
/// model selections produce distinct cache lines — a query embedded
/// under nomic-embed-text-v2-moe must NOT collide with the same
/// query under nomic-embed-text v1.
#[derive(Eq, PartialEq, Hash, Clone)]
struct EmbedCacheKey {
model: String,
text: String,
}
/// Default LRU cache size — 4096 entries × ~6KB per 768-d f64
/// vector ≈ 24MB. Sized for typical staffing-domain repetition
/// (coordinator workflows have query repetition rates around 70-90%
/// per session). Tunable via [aibridge].embed_cache_size in the
/// config; 0 disables the cache entirely.
const DEFAULT_EMBED_CACHE_SIZE: usize = 4096;
#[derive(Clone)] #[derive(Clone)]
pub struct AiClient { pub struct AiClient {
client: Client, client: Client,
base_url: String, base_url: String,
gateway_url: Option<String>, gateway_url: Option<String>,
/// Closes the 63× perf gap with Go side. Mirrors the shape of
/// Go's internal/embed/cached.go::CachedProvider — same
/// (model, text) → vector caching, same nil-disable semantics.
/// None = caching disabled (cache_size=0); Some = bounded LRU.
embed_cache: Option<Arc<Mutex<LruCache<EmbedCacheKey, Vec<f64>>>>>,
/// Hit / miss counters for /admin observability + load-test
/// validation. Atomic so Clone'd AiClients share the same counts.
embed_cache_hits: Arc<AtomicU64>,
embed_cache_misses: Arc<AtomicU64>,
/// Pinned at construction time so the EmbedResponse can carry
/// dimension consistently even when every text was a cache hit
/// (no fresh sidecar call to learn the dim from). Set on first
/// successful sidecar embed; checked on every cache hit.
cached_dim: Arc<AtomicU64>,
} }
// -- Request/Response types -- // -- Request/Response types --
@@ -95,17 +131,49 @@ pub struct RerankResponse {
impl AiClient { impl AiClient {
pub fn new(base_url: &str) -> Self { pub fn new(base_url: &str) -> Self {
Self::with_embed_cache(base_url, DEFAULT_EMBED_CACHE_SIZE)
}
/// Constructs an AiClient with an explicit embed-cache size.
/// Pass 0 to disable the cache entirely (matches Go-side
/// CachedProvider's nil-cache semantics).
pub fn with_embed_cache(base_url: &str, cache_size: usize) -> Self {
let client = Client::builder() let client = Client::builder()
.timeout(Duration::from_secs(120)) .timeout(Duration::from_secs(120))
.build() .build()
.expect("failed to build HTTP client"); .expect("failed to build HTTP client");
let embed_cache = if cache_size > 0 {
// SAFETY: cache_size > 0 just verified, NonZeroUsize::new
// returns Some.
let cap = NonZeroUsize::new(cache_size).expect("cache_size > 0");
Some(Arc::new(Mutex::new(LruCache::new(cap))))
} else {
None
};
Self { Self {
client, client,
base_url: base_url.trim_end_matches('/').to_string(), base_url: base_url.trim_end_matches('/').to_string(),
gateway_url: None, gateway_url: None,
embed_cache,
embed_cache_hits: Arc::new(AtomicU64::new(0)),
embed_cache_misses: Arc::new(AtomicU64::new(0)),
cached_dim: Arc::new(AtomicU64::new(0)),
} }
} }
/// Cache hit/miss/size snapshot. Useful for /admin endpoints +
/// load-test validation ("did the cache fire as expected?").
pub fn embed_cache_stats(&self) -> (u64, u64, usize) {
let hits = self.embed_cache_hits.load(Ordering::Relaxed);
let misses = self.embed_cache_misses.load(Ordering::Relaxed);
let len = self
.embed_cache
.as_ref()
.map(|c| c.lock().map(|g| g.len()).unwrap_or(0))
.unwrap_or(0);
(hits, misses, len)
}
/// Same as `new`, but every `generate()` is routed through /// Same as `new`, but every `generate()` is routed through
/// `${gateway_url}/v1/chat` (provider=ollama) for observability. /// `${gateway_url}/v1/chat` (provider=ollama) for observability.
/// Use this for callers OUTSIDE the gateway. Inside the gateway /// Use this for callers OUTSIDE the gateway. Inside the gateway
@@ -127,10 +195,104 @@ impl AiClient {
resp.json().await.map_err(|e| format!("invalid response: {e}")) resp.json().await.map_err(|e| format!("invalid response: {e}"))
} }
/// Embed with per-text LRU caching. Mirrors Go-side
/// CachedProvider behavior: cache key is (model, text);
/// cache-hit texts skip the sidecar; cache-miss texts batch
/// into a single sidecar call; results are interleaved in the
/// caller's input order.
///
/// Closes ~95% of the load-test perf gap vs Go side (loadgen
/// 2026-05-01: Rust 128 RPS → with cache ≥ 7000 RPS expected
/// for warm-cache workloads). Cold-cache behavior unchanged
/// (every text is a miss → single sidecar call, identical to
/// pre-cache).
pub async fn embed(&self, req: EmbedRequest) -> Result<EmbedResponse, String> { pub async fn embed(&self, req: EmbedRequest) -> Result<EmbedResponse, String> {
let model_key = req.model.clone().unwrap_or_default();
// Fast path: cache disabled → original behavior.
let Some(cache) = self.embed_cache.as_ref() else {
return self.embed_uncached(&req).await;
};
if req.texts.is_empty() {
return self.embed_uncached(&req).await;
}
// First pass: check cache for each text. Track which positions
// need a sidecar fetch.
let mut embeddings: Vec<Option<Vec<f64>>> = vec![None; req.texts.len()];
let mut miss_indices: Vec<usize> = Vec::new();
let mut miss_texts: Vec<String> = Vec::new();
{
let mut guard = cache.lock().map_err(|e| format!("cache lock poisoned: {e}"))?;
for (i, text) in req.texts.iter().enumerate() {
let key = EmbedCacheKey { model: model_key.clone(), text: text.clone() };
if let Some(vec) = guard.get(&key) {
embeddings[i] = Some(vec.clone());
self.embed_cache_hits.fetch_add(1, Ordering::Relaxed);
} else {
miss_indices.push(i);
miss_texts.push(text.clone());
self.embed_cache_misses.fetch_add(1, Ordering::Relaxed);
}
}
}
// All hit? Return immediately. Use cached_dim to populate
// the response dimension (no sidecar to ask).
if miss_indices.is_empty() {
let dim = self.cached_dim.load(Ordering::Relaxed) as usize;
let dim = if dim == 0 { embeddings[0].as_ref().map(|v| v.len()).unwrap_or(0) } else { dim };
return Ok(EmbedResponse {
embeddings: embeddings.into_iter().map(|opt| opt.expect("filled")).collect(),
model: req.model.unwrap_or_else(|| "nomic-embed-text".to_string()),
dimensions: dim,
});
}
// Second pass: fetch the misses in one sidecar call.
let miss_req = EmbedRequest { texts: miss_texts.clone(), model: req.model.clone() };
let resp = self.embed_uncached(&miss_req).await?;
if resp.embeddings.len() != miss_texts.len() {
return Err(format!(
"embed cache: sidecar returned {} embeddings for {} texts",
resp.embeddings.len(),
miss_texts.len()
));
}
// Pin cached_dim on first successful response.
if resp.dimensions > 0 {
self.cached_dim.store(resp.dimensions as u64, Ordering::Relaxed);
}
// Insert misses into cache + fill response slots.
{
let mut guard = cache.lock().map_err(|e| format!("cache lock poisoned: {e}"))?;
for (j, idx) in miss_indices.iter().enumerate() {
let key = EmbedCacheKey {
model: model_key.clone(),
text: miss_texts[j].clone(),
};
let vec = resp.embeddings[j].clone();
guard.put(key, vec.clone());
embeddings[*idx] = Some(vec);
}
}
Ok(EmbedResponse {
embeddings: embeddings.into_iter().map(|opt| opt.expect("filled")).collect(),
model: resp.model,
dimensions: resp.dimensions,
})
}
/// Direct sidecar call — original pre-cache behavior. Used
/// internally by embed() for cache-miss batches and as the
/// transparent fallback when the cache is disabled.
async fn embed_uncached(&self, req: &EmbedRequest) -> Result<EmbedResponse, String> {
let resp = self.client let resp = self.client
.post(format!("{}/embed", self.base_url)) .post(format!("{}/embed", self.base_url))
.json(&req) .json(req)
.send() .send()
.await .await
.map_err(|e| format!("embed request failed: {e}"))?; .map_err(|e| format!("embed request failed: {e}"))?;
crates/gateway/src/execution_loop/mod.rs
+24 -15
View File
@@ -582,10 +582,10 @@ impl ExecutionLoop {
/// Phase 20 step (8) — T3 overseer escalation. /// Phase 20 step (8) — T3 overseer escalation.
/// ///
/// When the local executor/reviewer loop can't self-correct, call /// When the local executor/reviewer loop can't self-correct, call
/// the cloud overseer (`gpt-oss:120b` via Ollama Cloud) with (a) /// the cloud overseer (`claude-opus-4-7` via OpenCode Zen) with
/// the KB context — recent outcomes + prior corrections for this /// (a) the KB context — recent outcomes + prior corrections for
/// sig_hash + task_class, across every profile that has run it — /// this sig_hash + task_class, across every profile that has run
/// and (b) the recent log tail. Its output is appended as a /// it — and (b) the recent log tail. Its output is appended as a
/// `system` role turn so the next executor generation sees it, /// `system` role turn so the next executor generation sees it,
/// AND written to `data/_kb/overseer_corrections.jsonl` so every /// AND written to `data/_kb/overseer_corrections.jsonl` so every
/// future profile activation reads from the same learning pool. /// future profile activation reads from the same learning pool.
@@ -593,9 +593,16 @@ impl ExecutionLoop {
/// This is the "pipe to the overviewer" piece from 2026-04-23 — /// This is the "pipe to the overviewer" piece from 2026-04-23 —
/// the overseer is now a first-class KB consumer AND producer, not /// the overseer is now a first-class KB consumer AND producer, not
/// a one-shot correction oracle. /// a one-shot correction oracle.
///
/// 2026-04-28: routed through OpenCode (Zen tier) for Claude Opus
/// 4.7. Frontier reasoning matters here because the overseer fires
/// only after local self-correction has failed twice — by that
/// point we need the strongest reasoning available, not the
/// cheapest token. Frequency is low so the Zen pay-per-token cost
/// stays bounded.
async fn escalate_to_overseer(&mut self, turn: u32, reason: &str) -> Result<(), String> { async fn escalate_to_overseer(&mut self, turn: u32, reason: &str) -> Result<(), String> {
let Some(cloud_key) = self.state.ollama_cloud_key.clone() else { let Some(opencode_key) = self.state.opencode_key.clone() else {
return Err("OLLAMA_CLOUD_KEY not configured — skipping escalation".into()); return Err("OPENCODE_API_KEY not configured — skipping escalation".into());
}; };
let kb = KbContext::load_for(&sig_hash(&self.req), &self.req.task_class).await; let kb = KbContext::load_for(&sig_hash(&self.req), &self.req.task_class).await;
@@ -604,16 +611,18 @@ impl ExecutionLoop {
let started = std::time::Instant::now(); let started = std::time::Instant::now();
let start_time = chrono::Utc::now(); let start_time = chrono::Utc::now();
let chat_req = crate::v1::ChatRequest { let chat_req = crate::v1::ChatRequest {
model: "gpt-oss:120b".to_string(), model: "claude-opus-4-7".to_string(),
messages: vec![crate::v1::Message::new_text("user", prompt.clone())], messages: vec![crate::v1::Message::new_text("user", prompt.clone())],
temperature: Some(0.1), temperature: Some(0.1),
max_tokens: None, max_tokens: None,
stream: Some(false), stream: Some(false),
think: Some(true), // overseer KEEPS thinking (Phase 20 rule) // Anthropic models on opencode reject `think` (handled in
provider: Some("ollama_cloud".into()), // the adapter), but we keep the intent flag for parity.
think: Some(true),
provider: Some("opencode".into()),
}; };
let resp = crate::v1::ollama_cloud::chat(&cloud_key, &chat_req).await let resp = crate::v1::opencode::chat(&opencode_key, &chat_req).await
.map_err(|e| format!("ollama_cloud: {e}"))?; .map_err(|e| format!("opencode: {e}"))?;
let latency_ms = started.elapsed().as_millis() as u64; let latency_ms = started.elapsed().as_millis() as u64;
let end_time = chrono::Utc::now(); let end_time = chrono::Utc::now();
let correction_text: String = resp.choices.into_iter().next() let correction_text: String = resp.choices.into_iter().next()
@@ -633,8 +642,8 @@ impl ExecutionLoop {
if let Some(lf) = &self.state.langfuse { if let Some(lf) = &self.state.langfuse {
use crate::v1::langfuse_trace::ChatTrace; use crate::v1::langfuse_trace::ChatTrace;
lf.emit_chat(ChatTrace { lf.emit_chat(ChatTrace {
provider: "ollama_cloud".into(), provider: "opencode".into(),
model: "gpt-oss:120b".into(), model: "claude-opus-4-7".into(),
input: vec![crate::v1::Message::new_text("user", prompt.clone())], input: vec![crate::v1::Message::new_text("user", prompt.clone())],
output: correction_text.clone(), output: correction_text.clone(),
prompt_tokens: resp.usage.prompt_tokens, prompt_tokens: resp.usage.prompt_tokens,
@@ -650,7 +659,7 @@ impl ExecutionLoop {
// Append to the transcript so the next executor turn sees it. // Append to the transcript so the next executor turn sees it.
self.append(LogEntry::new( self.append(LogEntry::new(
turn, "system", "gpt-oss:120b", "overseer_correction", turn, "system", "claude-opus-4-7", "overseer_correction",
serde_json::json!({ serde_json::json!({
"reason": reason, "reason": reason,
"correction": correction_text, "correction": correction_text,
@@ -672,7 +681,7 @@ impl ExecutionLoop {
"task_class": self.req.task_class, "task_class": self.req.task_class,
"operation": self.req.operation, "operation": self.req.operation,
"reason": reason, "reason": reason,
"model": "gpt-oss:120b", "model": "claude-opus-4-7",
"correction": correction_text, "correction": correction_text,
"applied_at_turn": turn, "applied_at_turn": turn,
"kb_context_used": kb, "kb_context_used": kb,
crates/vectord/src/rag.rs
+6 -2
View File
@@ -163,7 +163,11 @@ pub async fn query(
// production caller of the Phase 21 primitives — see audit finding // production caller of the Phase 21 primitives — see audit finding
// "Phase 21 Rust primitives are wired but not CALLED by any // "Phase 21 Rust primitives are wired but not CALLED by any
// production surface" from 2026-04-21. // production surface" from 2026-04-21.
let mut cont_opts = ContinuableOpts::new("qwen2.5:latest"); // 2026-04-30 model bump: qwen2.5:latest → qwen3.5:latest to match
// the small-model-pipeline local-tier default. Same JSON-clean
// property, more capacity. think=Some(false) preserved — RAG hot
// path doesn't need reasoning traces; direct answers only.
let mut cont_opts = ContinuableOpts::new("qwen3.5:latest");
cont_opts.max_tokens = Some(512); cont_opts.max_tokens = Some(512);
cont_opts.temperature = Some(0.2); cont_opts.temperature = Some(0.2);
cont_opts.shape = ResponseShape::Text; cont_opts.shape = ResponseShape::Text;
@@ -176,7 +180,7 @@ pub async fn query(
// echoes whatever Ollama loaded). Use the configured tier model // echoes whatever Ollama loaded). Use the configured tier model
// for now; if RAG needs to report the actual resolved model, // for now; if RAG needs to report the actual resolved model,
// the runner can add a post-call ps probe later. // the runner can add a post-call ps probe later.
model: "qwen2.5:latest".to_string(), model: "qwen3.5:latest".to_string(),
sources: results, sources: results,
tokens_generated: None, tokens_generated: None,
}) })
data/headshots/manifest.jsonl
+1000 -198
View File
File diff suppressed because it is too large Load Diff
lakehouse.toml
+11 -4
View File
@@ -48,8 +48,13 @@ url = "http://localhost:3200"
[ai] [ai]
embed_model = "nomic-embed-text" embed_model = "nomic-embed-text"
gen_model = "qwen2.5" # Local-tier defaults bumped 2026-04-30: qwen3.5:latest is the
rerank_model = "qwen2.5" # stronger local rung in the 5-loop substrate (per
# project_small_model_pipeline_vision.md). Same JSON-clean property
# as qwen2.5, more capacity. Ollama still serves both — bump back
# in this file if a workload regressed.
gen_model = "qwen3.5:latest"
rerank_model = "qwen3.5:latest"
[auth] [auth]
enabled = false enabled = false
@@ -72,7 +77,9 @@ min_recall = 0.9 # never promote below this
max_trials_per_hour = 20 # hard budget cap max_trials_per_hour = 20 # hard budget cap
# Model roster — available for profile hot-swap # Model roster — available for profile hot-swap
# qwen3.5:latest: stronger local rung — JSON-clean, 8K+ context,
# default for gen_model and rerank_model
# qwen3: 8.2B, 40K context, thinking+tools, best for reasoning tasks # qwen3: 8.2B, 40K context, thinking+tools, best for reasoning tasks
# qwen2.5: 7B, 8K context, fast, good for SQL generation # qwen2.5: 7B, 8K context, fast — kept loaded for the 2026-04 era
# mistral: 7B, 8K context, good for general generation # comparison runs; new defaults use qwen3.5:latest
# nomic-embed-text: 137M, embedding-only, used by all profiles # nomic-embed-text: 137M, embedding-only, used by all profiles
mcp-server/console.html
+39 -16
View File
@@ -56,7 +56,12 @@ details .body{padding-top:10px;font-size:12px;color:#8b949e}
.worker{display:flex;align-items:center;gap:10px;padding:8px 10px;background:#161b22;border-radius:6px;margin-bottom:4px;font-size:12px;border-left:3px solid #30363d} .worker{display:flex;align-items:center;gap:10px;padding:8px 10px;background:#161b22;border-radius:6px;margin-bottom:4px;font-size:12px;border-left:3px solid #30363d}
.worker .av{width:32px;height:32px;border-radius:50%;background:#0d1117;border:1px solid #21262d;display:flex;align-items:center;justify-content:center;font-weight:600;color:#c9d1d9;font-size:11px;flex-shrink:0;letter-spacing:0.5px;overflow:hidden;position:relative} .worker .av{width:32px;height:32px;border-radius:50%;background:#0d1117;border:1px solid #21262d;display:flex;align-items:center;justify-content:center;font-weight:600;color:#c9d1d9;font-size:11px;flex-shrink:0;letter-spacing:0.5px;overflow:hidden;position:relative}
.worker .av img{position:absolute;inset:0;width:100%;height:100%;object-fit:cover;display:block} .worker .av img{position:absolute;inset:0;width:100%;height:100%;object-fit:cover;display:block;
/* Softening — mirror of search.html. Pulls saturation + contrast off
the SDXL Turbo over-render so faces feel less "AI-generated".
If you tweak one, tweak the other. */
filter: saturate(0.86) contrast(0.93) brightness(1.02) blur(0.3px);
}
.worker[data-role-band="warehouse"]{border-left-color:#58a6ff} .worker[data-role-band="warehouse"]{border-left-color:#58a6ff}
.worker[data-role-band="production"]{border-left-color:#d29922} .worker[data-role-band="production"]{border-left-color:#d29922}
.worker[data-role-band="trades"]{border-left-color:#bc8cff} .worker[data-role-band="trades"]{border-left-color:#bc8cff}
@@ -262,6 +267,37 @@ var NAMES_EAST_ASIAN_C=new Set(['Wei','Mei','Yi','Jin','Chen','Lin','Liu','Wang'
var NAMES_HISPANIC_C=new Set(['Carmen','Carlos','Maria','Diego','Hector','Jorge','Julio','Manuel','Miguel','Pedro','Raul','Ricardo','Roberto','Sergio','Antonio','Esperanza','Luz','Sofia','Lucia','Isabella','Camila','Valentina','Mariana','Elena','Rosa','Catalina','Esteban','Fernando','Eduardo','Javier','Alejandro','Andres','Mateo','Santiago','Sebastian','Emilio','Tomas','Cristina','Daniela','Gabriela','Ximena','Adriana','Beatriz','Pilar','Mercedes','Xavier','Marisol','Guadalupe','Lupita','Inez','Itzel','Yesenia','Joaquin','Ignacio','Rafael','Salvador','Cesar','Arturo','Armando','Hugo','Marco','Alejandra','Felipe','Gerardo','Jaime','Leonardo','Luis','Pablo','Ramon']); var NAMES_HISPANIC_C=new Set(['Carmen','Carlos','Maria','Diego','Hector','Jorge','Julio','Manuel','Miguel','Pedro','Raul','Ricardo','Roberto','Sergio','Antonio','Esperanza','Luz','Sofia','Lucia','Isabella','Camila','Valentina','Mariana','Elena','Rosa','Catalina','Esteban','Fernando','Eduardo','Javier','Alejandro','Andres','Mateo','Santiago','Sebastian','Emilio','Tomas','Cristina','Daniela','Gabriela','Ximena','Adriana','Beatriz','Pilar','Mercedes','Xavier','Marisol','Guadalupe','Lupita','Inez','Itzel','Yesenia','Joaquin','Ignacio','Rafael','Salvador','Cesar','Arturo','Armando','Hugo','Marco','Alejandra','Felipe','Gerardo','Jaime','Leonardo','Luis','Pablo','Ramon']);
var NAMES_BLACK_C=new Set(['DeShawn','Jamal','Aisha','Latoya','Tyrone','Malik','Imani','Keisha','Tariq','Lakisha','Kenya','Tamika','Andre','Marcus','Demetrius','Jermaine','Reggie','Tyrese','Darius','Trevon','Kareem','Damon','Jalen','Jaylen','Dwayne','DaQuan','Aaliyah','Kiara','Janelle','Jasmine','Tanisha','Maurice','Tyrell','Kwame','Khalil','Terrell','Cedric','Nia','Zuri','Jada','Ebony','Dominique']); var NAMES_BLACK_C=new Set(['DeShawn','Jamal','Aisha','Latoya','Tyrone','Malik','Imani','Keisha','Tariq','Lakisha','Kenya','Tamika','Andre','Marcus','Demetrius','Jermaine','Reggie','Tyrese','Darius','Trevon','Kareem','Damon','Jalen','Jaylen','Dwayne','DaQuan','Aaliyah','Kiara','Janelle','Jasmine','Tanisha','Maurice','Tyrell','Kwame','Khalil','Terrell','Cedric','Nia','Zuri','Jada','Ebony','Dominique']);
var NAMES_MIDDLE_EASTERN_C=new Set(['Layla','Omar','Khalid','Fatima','Yasmin','Hassan','Hussein','Ahmed','Mohamed','Mohammed','Ali','Karim','Yusuf','Yara','Nadia','Zainab','Rania','Samira','Mariam','Salma','Ibrahim','Mahmoud','Saif','Anwar','Bilal','Faisal','Hamza','Imran','Sami','Wael','Zaid','Amira','Iman','Lina','Mona','Noor','Rana','Soha','Zara']); var NAMES_MIDDLE_EASTERN_C=new Set(['Layla','Omar','Khalid','Fatima','Yasmin','Hassan','Hussein','Ahmed','Mohamed','Mohammed','Ali','Karim','Yusuf','Yara','Nadia','Zainab','Rania','Samira','Mariam','Salma','Ibrahim','Mahmoud','Saif','Anwar','Bilal','Faisal','Hamza','Imran','Sami','Wael','Zaid','Amira','Iman','Lina','Mona','Noor','Rana','Soha','Zara']);
// Surname → ethnicity. Surname is more diagnostic than first name
// for hispanic and asian — "Anna Cruz" is hispanic via surname.
var SURNAMES_HISPANIC_C=new Set(['Garcia','Rodriguez','Martinez','Hernandez','Lopez','Gonzalez','Perez','Sanchez','Ramirez','Torres','Flores','Rivera','Gomez','Diaz','Reyes','Cruz','Morales','Ortiz','Gutierrez','Chavez','Ramos','Ruiz','Alvarez','Mendoza','Vasquez','Castillo','Jimenez','Moreno','Romero','Herrera','Medina','Aguilar','Vargas','Castro','Fernandez','Guzman','Munoz','Salazar','Ortega','Delgado','Estrada','Ayala','Pena','Cabrera','Alvarado','Espinoza','Padilla','Cardenas','Cortes','Ibarra','Vega','Soto','Lara','Navarro','Campos','Acosta','Rios','Marquez','Sandoval','Maldonado','Solis','Rojas','Mejia','Beltran','Cervantes','Lozano','Carrillo','Trevino','Robles','Tapia','Lugo']);
var SURNAMES_SOUTH_ASIAN_C=new Set(['Patel','Singh','Kumar','Sharma','Gupta','Shah','Mehta','Desai','Joshi','Reddy','Nair','Iyer','Verma','Agarwal','Kapoor','Chopra','Malhotra','Banerjee','Chatterjee','Mukherjee','Das','Sen','Bose','Roy','Sinha','Trivedi','Pandey','Mishra','Tiwari','Yadav','Chauhan','Rana','Thakur','Pillai','Menon','Krishnan','Rao','Naidu','Pradhan','Acharya','Devi','Kaur']);
var SURNAMES_EAST_ASIAN_C=new Set(['Chen','Wang','Li','Liu','Yang','Huang','Zhao','Wu','Zhou','Xu','Zhu','Sun','Ma','Lin','Lee','Kim','Park','Choi','Jung','Kang','Cho','Yoon','Han','Lim','Oh','Nakamura','Tanaka','Suzuki','Yamamoto','Sato','Watanabe','Takahashi','Kobayashi','Yoshida','Saito','Nguyen','Tran','Le','Pham','Hoang','Phan','Vu','Vo','Dang','Bui','Do','Ngo','Truong','Mai','Cao','Wong','Tang','Tan','Cheng','Lau','Leung','Ng','Cheung','Yip','Hsu','Tsai','Hsieh']);
var SURNAMES_MIDDLE_EASTERN_C=new Set(['Khan','Ahmed','Hussein','Hassan','Ali','Mahmoud','Mohamed','Mohammed','Saleh','Aziz','Karim','Hamad','Najjar','Haddad','Khoury','Mansour','Rahman','Iqbal','Malik','Sheikh','Siddiqui','Qureshi','Saeed']);
function guessEthnicityFromName(first, last){
if(last){
var s=last.replace(/[^A-Za-z]/g,'');
if(s){
var sc=s[0].toUpperCase()+s.slice(1).toLowerCase();
if(SURNAMES_HISPANIC_C.has(sc)) return 'hispanic';
if(SURNAMES_MIDDLE_EASTERN_C.has(sc)) return 'middle_eastern';
if(SURNAMES_SOUTH_ASIAN_C.has(sc)) return 'south_asian';
if(SURNAMES_EAST_ASIAN_C.has(sc)) return 'east_asian';
}
}
if(first){
var clean=first.replace(/[^A-Za-z]/g,'');
if(clean){
var c=clean[0].toUpperCase()+clean.slice(1).toLowerCase();
if(NAMES_MIDDLE_EASTERN_C.has(c)) return 'middle_eastern';
if(NAMES_BLACK_C.has(c)) return 'black';
if(NAMES_HISPANIC_C.has(c)) return 'hispanic';
if(NAMES_SOUTH_ASIAN_C.has(c)) return 'south_asian';
if(NAMES_EAST_ASIAN_C.has(c)) return 'east_asian';
}
}
return 'caucasian';
}
function guessEthnicityFromFirstName(n){ function guessEthnicityFromFirstName(n){
if(!n) return 'caucasian'; if(!n) return 'caucasian';
var clean=n.replace(/[^A-Za-z]/g,''); if(!clean) return 'caucasian'; var clean=n.replace(/[^A-Za-z]/g,''); if(!clean) return 'caucasian';
@@ -281,21 +317,8 @@ function workerRow(name, role, detail, opts){
if(band.band) w.dataset.roleBand = band.band; if(band.band) w.dataset.roleBand = band.band;
var initials = (name||'?').split(' ').map(function(s){return (s[0]||'').toUpperCase()}).join('').substring(0,2); var initials = (name||'?').split(' ').map(function(s){return (s[0]||'').toUpperCase()}).join('').substring(0,2);
var av = el('div','av',initials); var av = el('div','av',initials);
// Real synthetic headshot via /headshots/<key>; deterministic so // Headshot insertion removed 2026-04-28. The .av element stays as
// same worker always gets the same face. Falls back to monogram if // a monogram-initials avatar.
// pool isn't fetched yet.
var faceKey = (opts.face_key) || name || '';
var firstName = (name||'').split(/\s+/)[0]||'';
var gHint = genderFor(firstName);
var eHint = guessEthnicityFromFirstName(firstName);
if(faceKey){
var img=document.createElement('img');
img.alt='';
img.loading='lazy';
img.src = P + '/headshots/' + encodeURIComponent(faceKey) + '?g='+gHint+'&e='+eHint;
img.onerror=function(){ this.remove(); };
av.appendChild(img);
}
w.appendChild(av); w.appendChild(av);
var info = el('div','info'); var info = el('div','info');
var nm = el('div','nm', name||'?'); var nm = el('div','nm', name||'?');
mcp-server/icon_recipes.ts
+123
View File
@@ -0,0 +1,123 @@
// Visual filler iconography rendered through ComfyUI. Distinct from
// role_scenes.ts (which renders portraits) — these are object/badge
// style renders that fill dead space on worker cards: cert pills,
// role-prop chips, hazard indicators, empty-state heroes.
//
// Layout on disk:
// data/icons_pool/{category}/{slug}.webp
//
// Cache invalidation:
// ICONS_VERSION mixes into the on-disk filename (slug includes
// version). Bump it after editing a recipe so prior renders are
// ignored on next view.
export type IconCategory = "cert" | "role_prop" | "status" | "hazard" | "empty";
export interface IconRecipe {
slug: string;
category: IconCategory;
// Text label that appears next to / under the icon. The front-end
// already renders this text in cert pills; the icon is supplementary.
display: string;
// Full diffusion prompt. Style guidance baked in. SDXL Turbo at 8
// steps reliably produces clean macro photography, so default to
// photographic prop shots over flat-vector illustrations (the model
// hallucinates noise into flat-vector geometry at low step counts).
prompt: string;
// Negative prompt — what NOT to render. Crucial for icons because
// SDXL likes to add hands/text/people unprompted.
negative?: string;
}
// Default negative prompt baked into every icon render unless the
// recipe overrides. Empirically, these terms are the top SDXL Turbo
// off-style failures.
export const DEFAULT_NEGATIVE =
"people, hands, faces, blurry, low quality, watermark, signature, "
+ "logos, copyright, distorted text, garbled letters, multiple objects";
// TODO J — review and tune the prompts here. Each one is what diffusion
// sees verbatim. The visual decision: photographic prop shots (macro
// photo of an actual badge / placard / sticker) vs flat-icon vector
// style. Default below is photographic — matches the worker headshot
// aesthetic. Flip a recipe to flat-vector by replacing "macro photograph"
// with "flat icon illustration on solid color background, minimal vector".
//
// Visual cues that work well in SDXL Turbo at 8 steps:
// - "macro photograph", "isolated on plain background", "studio lighting"
// - Concrete colors ("orange and black warning diamond") not adjectives
// - Avoid: small text in the prompt (model garbles it), specific brand
// names (creates fake logos), detailed scene composition
const CERT_ICONS: IconRecipe[] = [
{ slug: "osha-10", category: "cert", display: "OSHA-10",
prompt: "macro photograph of a circular yellow safety badge with a black hard hat icon at center, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "osha-30", category: "cert", display: "OSHA-30",
prompt: "macro photograph of a circular orange safety badge with a black hard hat icon at center, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "first-aid-cpr", category: "cert", display: "First Aid/CPR",
prompt: "macro photograph of a small enamel pin badge featuring a bold red cross on a white circular background, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "hazmat", category: "cert", display: "Hazmat",
prompt: "macro photograph of a HAZMAT warning placard, bold orange and black diamond shape with a flame icon, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "forklift", category: "cert", display: "Forklift",
prompt: "macro photograph of a yellow industrial forklift safety badge with a forklift silhouette icon, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "reach-truck", category: "cert", display: "Reach Truck",
prompt: "macro photograph of a navy blue industrial certification badge with a warehouse reach-truck silhouette icon, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "order-picker", category: "cert", display: "Order Picker",
prompt: "macro photograph of a green industrial certification badge with a warehouse order-picker silhouette icon, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "lockout-tagout", category: "cert", display: "Lockout/Tagout",
prompt: "macro photograph of a bright red padlock tag with a danger warning, hanging on a metal industrial valve, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "msds", category: "cert", display: "MSDS",
prompt: "macro photograph of a folded chemical safety data sheet booklet with chemical hazard pictograms visible on cover, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "confined-space", category: "cert", display: "Confined Space",
prompt: "macro photograph of a yellow confined space warning sign featuring a manhole entry icon, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "servsafe", category: "cert", display: "ServSafe",
prompt: "macro photograph of a dark green food safety certification badge featuring a stylized chef hat icon, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "fire-safety", category: "cert", display: "Fire Safety",
prompt: "macro photograph of a red enamel pin badge featuring a flame icon and a fire extinguisher silhouette, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "iso-9001", category: "cert", display: "ISO 9001",
prompt: "macro photograph of a deep blue circular quality-management certification seal with embossed metallic ring, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
];
// Role-band visual chips — small icons that go in the role pill area.
// One per band, optional inline supplement to the existing colored pill.
const ROLE_PROP_ICONS: IconRecipe[] = [
{ slug: "warehouse", category: "role_prop", display: "Warehouse",
prompt: "macro photograph of a yellow hard hat with a high-visibility safety vest folded behind it, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "production", category: "role_prop", display: "Production",
prompt: "macro photograph of a navy blue work shirt and protective safety glasses on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "trades", category: "role_prop", display: "Trades",
prompt: "macro photograph of a leather work glove and a small adjustable wrench on a neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "driver", category: "role_prop", display: "Driver",
prompt: "macro photograph of a navy delivery driver baseball cap and a clipboard manifest on a neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "lead", category: "role_prop", display: "Lead",
prompt: "macro photograph of a tablet showing a bar chart and a high-vis vest folded beside it on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
];
export const ICONS: Record<string, IconRecipe> = Object.fromEntries(
[...CERT_ICONS, ...ROLE_PROP_ICONS].map((r) => [`${r.category}/${r.slug}`, r]),
);
// v2 — 256×256 canvas, intended to be displayed monochrome via CSS
// `filter: grayscale(1)`. Smaller canvas, tighter crops, crisper at
// 14px display size.
export const ICONS_VERSION = "v2";
// Map a free-form cert string from the data ("First Aid/CPR",
// "OSHA-10", "Lockout/Tagout") to the canonical slug used here.
// Returns null if no recipe matches.
export function certToSlug(cert: string): string | null {
const c = (cert || "").trim().toLowerCase().replace(/\s+/g, "-");
if (c === "osha-10") return "osha-10";
if (c === "osha-30") return "osha-30";
if (c.startsWith("first") || c.includes("cpr")) return "first-aid-cpr";
if (c === "hazmat" || c.startsWith("hazwoper")) return "hazmat";
if (c === "forklift" || c.startsWith("pit")) return "forklift";
if (c.startsWith("reach")) return "reach-truck";
if (c.startsWith("order")) return "order-picker";
if (c.startsWith("lockout") || c.includes("tagout")) return "lockout-tagout";
if (c === "msds" || c.startsWith("ghs")) return "msds";
if (c.startsWith("confined")) return "confined-space";
if (c === "servsafe") return "servsafe";
if (c.startsWith("fire")) return "fire-safety";
if (c.startsWith("iso")) return "iso-9001";
return null;
}
mcp-server/index.ts
+405 -22
View File
@@ -19,8 +19,17 @@ import { SSEServerTransport } from "@modelcontextprotocol/sdk/server/sse.js";
import { z } from "zod"; import { z } from "zod";
import { startTrace, logSpan, logGeneration, scoreTrace, flush as flushTraces } from "./tracing.js"; import { startTrace, logSpan, logGeneration, scoreTrace, flush as flushTraces } from "./tracing.js";
import { buildPermitBrief } from "./entity.js"; import { buildPermitBrief } from "./entity.js";
import { roleBand, SCENES, SCENES_VERSION, FACE_RENDER_DIM, type RoleBand } from "./role_scenes.js";
import { ICONS, ICONS_VERSION, DEFAULT_NEGATIVE, certToSlug, type IconRecipe } from "./icon_recipes.js";
const BASE = process.env.LAKEHOUSE_URL || "http://localhost:3100"; const BASE = process.env.LAKEHOUSE_URL || "http://localhost:3100";
// G5 cutover prep (2026-05-01): when GO_LAKEHOUSE_URL is set, the
// `/_go/*` pass-through routes to the Go gateway. Off-by-default
// (empty = disabled, returns 503 on /_go/*). Reversible via unset.
// Doesn't modify any existing tool — additive parallel path so
// operators can validate Go-side handlers under real Bun-frontend
// shape without touching the production Rust path.
const GO_BASE = process.env.GO_LAKEHOUSE_URL || "";
const PORT = parseInt(process.env.MCP_PORT || "3700"); const PORT = parseInt(process.env.MCP_PORT || "3700");
// ─── Staffer roster — used by the per-staffer hot-swap index (G). ──── // ─── Staffer roster — used by the per-staffer hot-swap index (G). ────
@@ -311,9 +320,9 @@ ${(buckets as any[] || []).map((b: any) => `- ${b.name}: ${b.backend} (${b.reach
- Ollama: :11434 - Ollama: :11434
## Available Models ## Available Models
- qwen3.5:latest: stronger local rung, JSON-clean (default for gen + rerank)
- qwen3: 8.2B, 40K context, thinking+tools (best for reasoning) - qwen3: 8.2B, 40K context, thinking+tools (best for reasoning)
- qwen2.5: 7B, 8K context (best for fast SQL generation) - qwen2.5: 7B, 8K context (legacy — 2026-04 era comparison runs only)
- mistral: 7B, 8K context (general generation)
- nomic-embed-text: 137M (embedding, automatic) - nomic-embed-text: 137M (embedding, automatic)
`; `;
return { contents: [{ uri: uri.href, mimeType: "text/plain", text }] }; return { contents: [{ uri: uri.href, mimeType: "text/plain", text }] };
@@ -709,6 +718,26 @@ async function main() {
return new Response(await r.text(), { status: r.status, headers: { "Content-Type": "application/json" } }); return new Response(await r.text(), { status: r.status, headers: { "Content-Type": "application/json" } });
} }
// G5 cutover slice: pass-through to GO_BASE (Go gateway). Same
// shape as /api/* but points at the Go side. Returns 503 when
// GO_LAKEHOUSE_URL isn't set so callers know the cutover slice
// is off rather than silently routing to Rust. No body
// transformation — caller responsible for sending Go-shaped
// requests (e.g. /v1/embed not /ai/embed).
if (url.pathname.startsWith("/_go/")) {
if (!GO_BASE) {
return err("GO_LAKEHOUSE_URL not set; /_go/* cutover slice is disabled", 503);
}
const path = url.pathname.replace("/_go", "");
const body = req.method !== "GET" ? await req.text() : undefined;
try {
const r = await fetch(`${GO_BASE}${path}`, { method: req.method, headers: { "Content-Type": "application/json" }, body });
return new Response(await r.text(), { status: r.status, headers: { "Content-Type": "application/json" } });
} catch (e) {
return err(`Go gateway unreachable at ${GO_BASE}: ${e}`, 502);
}
}
// Proof — narrative HTML served from mcp-server/proof.html. // Proof — narrative HTML served from mcp-server/proof.html.
// Live tests consumed client-side via /proof.json. // Live tests consumed client-side via /proof.json.
if (url.pathname === "/proof") { if (url.pathname === "/proof") {
@@ -1225,6 +1254,191 @@ async function main() {
// OSHA national, Chicago history, ticker chart, parent link, // OSHA national, Chicago history, ticker chart, parent link,
// federal contracts, debarment, unions, training. Click any // federal contracts, debarment, unions, training. Click any
// contractor name in a permit Entity Brief to land here. // contractor name in a permit Entity Brief to land here.
// ComfyUI-generated portrait — every call is unique by (key,
// gender, race, age, role) tuple. First hit takes ~1.5s on
// the A4000; subsequent hits read from disk. Use this for
// contractor / profile modal where one worker gets the
// spotlight. NB: declared BEFORE the pool route so the prefix
// match doesn't intercept it.
// Single source of truth for the pre-render script. Read
// role_scenes.ts SCENES + SCENES_VERSION so a Python pre-render
// job (scripts/staffing/render_role_pool.py) builds the role-
// aware pool with the exact prompts the server will use on the
// ComfyUI hot-path. No drift.
if (url.pathname === "/headshots/_scenes" && req.method === "GET") {
return Response.json({ version: SCENES_VERSION, scenes: SCENES });
}
// Single source of truth for icon_recipes.ts. Used by the
// pre-render script (scripts/staffing/render_icons.py) and any
// tooling that wants to enumerate available icons.
if (url.pathname === "/icons/_recipes" && req.method === "GET") {
return Response.json({
version: ICONS_VERSION,
default_negative: DEFAULT_NEGATIVE,
recipes: ICONS,
});
}
// Free-text cert resolver: front-end passes the raw cert string
// from the data ("First Aid/CPR", "OSHA-10", "Lockout/Tagout")
// and we resolve to a recipe slug + 302 to the cached/rendered
// icon. Returns 404 (not error) when no recipe matches — the
// front-end can hang an `onerror="this.remove()"` to silently
// drop the img tag for unrecognized certs.
if (url.pathname === "/icons/cert" && req.method === "GET") {
const text = url.searchParams.get("text") || "";
const slug = certToSlug(text);
if (!slug) return new Response(`no recipe for cert: ${text}`, { status: 404 });
return new Response(null, {
status: 302,
headers: { "Location": `/icons/render/cert/${slug}` },
});
}
// Cert / role-prop / status / hazard / empty icons. Lookup is
// category/slug; on cache miss the route renders via ComfyUI.
// Filename layout: data/icons_pool/{category}/{slug}_{version}.webp
// — the version suffix means editing a recipe yields a new file
// rather than overwriting in place, so a misfire is recoverable.
if (url.pathname.startsWith("/icons/render/") && req.method === "GET") {
const rest = url.pathname.slice("/icons/render/".length);
const recipe: IconRecipe | undefined = ICONS[rest];
if (!recipe) return new Response(`unknown icon: ${rest}`, { status: 404 });
const ICONS_DIR = "/home/profit/lakehouse/data/icons_pool";
await Bun.$`mkdir -p ${ICONS_DIR}/${recipe.category}`.quiet();
const cachePath = `${ICONS_DIR}/${recipe.category}/${recipe.slug}_${ICONS_VERSION}.webp`;
const cached = Bun.file(cachePath);
if (await cached.exists()) {
return new Response(cached, {
headers: {
"Content-Type": "image/webp",
"Cache-Control": "public, max-age=86400",
"X-Icon-Source": "cached",
"X-Icon-Recipe": recipe.slug,
},
});
}
// Deterministic seed per recipe — same recipe always renders
// the same icon. Mixing the version means SCENES_VERSION-
// style invalidation works for icons too.
const seedStr = `${recipe.category}|${recipe.slug}|${ICONS_VERSION}`;
let seed = 5381;
for (let i = 0; i < seedStr.length; i++) seed = ((seed << 5) + seed + seedStr.charCodeAt(i)) | 0;
seed = Math.abs(seed) % 2147483647;
try {
const genResp = await fetch("http://localhost:3600/generate", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
prompt: recipe.prompt,
negative_prompt: recipe.negative ?? DEFAULT_NEGATIVE,
// 256×256 — smaller canvas = cleaner icon. SDXL Turbo
// at 8 steps adds visible texture/noise into 512² that
// looks "AI" at small display sizes; tightening to 256
// both renders ~3× faster and produces crisper edges
// when the front-end downsamples to 14px.
width: 256,
height: 256,
steps: 8,
seed,
}),
signal: AbortSignal.timeout(30000),
});
if (!genResp.ok) return new Response(`gen failed: ${genResp.status}`, { status: 502 });
const data: any = await genResp.json();
if (!data.image) return new Response("no image returned", { status: 502 });
const bytes = Uint8Array.from(atob(data.image), (c) => c.charCodeAt(0));
await Bun.write(cachePath, bytes);
return new Response(bytes, {
headers: {
"Content-Type": "image/webp",
"Cache-Control": "public, max-age=86400",
"X-Icon-Source": "fresh",
"X-Icon-Recipe": recipe.slug,
"X-Icon-Gen-Ms": String(data.time_ms || 0),
},
});
} catch (e: any) {
return new Response(`gen error: ${e.message}`, { status: 502 });
}
}
if (url.pathname.startsWith("/headshots/generate/") && req.method === "GET") {
const key = decodeURIComponent(url.pathname.slice("/headshots/generate/".length));
if (!key) return new Response("missing key", { status: 400 });
const g = (url.searchParams.get("g") || "person").toLowerCase();
const r = (url.searchParams.get("e") || "").toLowerCase();
const role = (url.searchParams.get("role") || "warehouse worker").toLowerCase();
const age = parseInt(url.searchParams.get("age") || "32", 10) || 32;
const band = roleBand(role);
// SCENES_VERSION mixes into the cache key so editing
// role_scenes.ts auto-invalidates prior renders — coordinator
// tweaks the warehouse prompt, every warehouse face refreshes
// on next view.
const cacheKey = await crypto.subtle.digest(
"SHA-256",
new TextEncoder().encode(`${key}|${g}|${r}|${role}|${age}|${SCENES_VERSION}`)
).then((b) => Array.from(new Uint8Array(b)).map((x) => x.toString(16).padStart(2, "0")).join("").slice(0, 24));
const GEN_DIR = "/home/profit/lakehouse/data/headshots_gen";
await Bun.$`mkdir -p ${GEN_DIR}`.quiet();
const cachePath = `${GEN_DIR}/${cacheKey}.webp`;
const cached = Bun.file(cachePath);
if (await cached.exists()) {
return new Response(cached, {
headers: {
"Content-Type": "image/webp",
"Cache-Control": "public, max-age=86400, immutable",
"X-Headshot-Source": "comfyui-cached",
},
});
}
const raceText = r === "hispanic" ? "Hispanic"
: r === "black" ? "Black"
: r === "south_asian" ? "South Asian"
: r === "east_asian" ? "East Asian"
: r === "middle_eastern" ? "Middle Eastern"
: "";
const genderText = g === "woman" ? "woman" : g === "man" ? "man" : "person";
const scene = SCENES[band].scene;
// Note: dropped "plain studio background" / "dslr" — those
// collapsed every render to interchangeable studio shots.
// The scene clause now carries clothing + backdrop so a
// forklift operator looks like a forklift operator.
const prompt = `professional headshot portrait of a ${age}-year-old ${raceText} ${genderText} ${role}, ${scene}, neutral confident expression, sharp focus, photorealistic`;
// Worker-derived seed — same input always picks the same
// pixel layout in StyleGAN2 latent space, so the face is
// deterministic per worker BUT distinct from any other
// worker that happens to share the same prompt. Without
// this, every (g, r, age, role) combo collapses to one face.
let seedHash = 0;
for (let i = 0; i < key.length; i++) seedHash = ((seedHash << 5) - seedHash + key.charCodeAt(i)) | 0;
const seed = Math.abs(seedHash) % 2147483647;
try {
const genResp = await fetch("http://localhost:3600/generate", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ prompt, width: FACE_RENDER_DIM, height: FACE_RENDER_DIM, steps: 8, seed }),
signal: AbortSignal.timeout(30000),
});
if (!genResp.ok) return new Response(`gen failed: ${genResp.status}`, { status: 502 });
const data: any = await genResp.json();
if (!data.image) return new Response("no image returned", { status: 502 });
const bytes = Uint8Array.from(atob(data.image), (c) => c.charCodeAt(0));
await Bun.write(cachePath, bytes);
return new Response(bytes, {
headers: {
"Content-Type": "image/webp",
"Cache-Control": "public, max-age=86400, immutable",
"X-Headshot-Source": "comfyui-fresh",
"X-Headshot-Gen-Ms": String(data.time_ms || 0),
},
});
} catch (e: any) {
return new Response(`gen error: ${e.message}`, { status: 502 });
}
}
// Headshot pool — synthetic StyleGAN faces from // Headshot pool — synthetic StyleGAN faces from
// thispersondoesnotexist.com fetched offline by // thispersondoesnotexist.com fetched offline by
// scripts/staffing/fetch_face_pool.py. Deterministic mapping: // scripts/staffing/fetch_face_pool.py. Deterministic mapping:
@@ -1249,19 +1463,47 @@ async function main() {
const raw = await Bun.file(`${HEADSHOT_DIR}/manifest.jsonl`).text(); const raw = await Bun.file(`${HEADSHOT_DIR}/manifest.jsonl`).text();
const lines = raw.trim().split("\n").filter(Boolean); const lines = raw.trim().split("\n").filter(Boolean);
const all = lines.map((l) => JSON.parse(l)); const all = lines.map((l) => JSON.parse(l));
// Build (gender × race) buckets so a request that names
// both narrows to the intersection. Missing intersections
// fall back to gender-only, then race-only, then all.
const byGR: Record<string, any[]> = {};
const byG: Record<string, any[]> = { man: [], woman: [] };
const byR: Record<string, any[]> = {};
// Filter excluded faces (e.g. minors) from every bucket
// and from the all-pool. They never get served.
const adults = all.filter((r: any) => !r.excluded);
for (const r of adults) {
if (r.gender === "man" || r.gender === "woman") byG[r.gender].push(r);
if (r.race) {
byR[r.race] = byR[r.race] || [];
byR[r.race].push(r);
if (r.gender === "man" || r.gender === "woman") {
const k = r.gender + "/" + r.race;
byGR[k] = byGR[k] || [];
byGR[k].push(r);
}
}
}
(globalThis as any)._faces = { (globalThis as any)._faces = {
all, all: adults,
man: all.filter((r: any) => r.gender === "man"), byG, byR, byGR,
woman: all.filter((r: any) => r.gender === "woman"), untagged: adults.filter((r: any) => !r.gender || (r.gender !== "man" && r.gender !== "woman")),
untagged: all.filter((r: any) => !r.gender || (r.gender !== "man" && r.gender !== "woman")), excluded_count: all.length - adults.length,
loaded_at: Date.now(), loaded_at: Date.now(),
}; };
if (key === "__reload") { if (key === "__reload") {
const byRSummary: Record<string, number> = {};
for (const k of Object.keys(byR)) byRSummary[k] = byR[k].length;
const byGRSummary: Record<string, number> = {};
for (const k of Object.keys(byGR)) byGRSummary[k] = byGR[k].length;
return Response.json({ return Response.json({
reloaded: true, reloaded: true,
total: all.length, total: all.length,
man: (globalThis as any)._faces.man.length, excluded: all.length - adults.length,
woman: (globalThis as any)._faces.woman.length, served_pool: adults.length,
by_gender: { man: byG.man.length, woman: byG.woman.length },
by_race: byRSummary,
by_gender_race: byGRSummary,
untagged: (globalThis as any)._faces.untagged.length, untagged: (globalThis as any)._faces.untagged.length,
}); });
} }
@@ -1269,20 +1511,84 @@ async function main() {
return new Response(`face pool not available: ${e.message}. Run scripts/staffing/fetch_face_pool.py first.`, { status: 503 }); return new Response(`face pool not available: ${e.message}. Run scripts/staffing/fetch_face_pool.py first.`, { status: 503 });
} }
} }
const F = (globalThis as any)._faces as { all: any[]; man: any[]; woman: any[]; untagged: any[] }; const F = (globalThis as any)._faces as {
all: any[];
byG: Record<string, any[]>;
byR: Record<string, any[]>;
byGR: Record<string, any[]>;
untagged: any[];
};
if (!F || !F.all.length) { if (!F || !F.all.length) {
return new Response("face pool empty", { status: 503 }); return new Response("face pool empty", { status: 503 });
} }
// Pool selection: gender hint > full pool. If no gender match, const wantRace = url.searchParams.get("e") || "";
// fall back to the full pool so the worker still gets a face.
// NOTE: role-aware pool + ComfyUI sparse redirect were removed
// 2026-04-28 — diffusion output at 8 steps with the existing
// editorial_hero workflow produced burnt-looking faces ("looks
// like someone burnt the pictures"). Until serve_imagegen.py
// is fixed to honor a portrait-friendly negative prompt and
// run with proper steps/cfg, every face comes from the studio
// pool (StyleGAN photos from thispersondoesnotexist.com) and
// gets B&W via CSS. The role pool files at
// data/headshots_role_pool/{v1,v2}/ stay on disk for when
// we can re-enable them.
// Studio pool only. Try gender×race intersection first, then
// fall back to gender-only or race-only if the intersection
// is sparse. Repeat faces are acceptable — better than
// serving the over-contrasty diffusion output.
let pool = F.all; let pool = F.all;
if (wantGender === "man" && F.man.length) pool = F.man; let bucket = "all";
else if (wantGender === "woman" && F.woman.length) pool = F.woman; if (wantGender && wantRace) {
const gr = F.byGR[wantGender + "/" + wantRace] || [];
if (gr.length > 0) {
// Use the intersection bucket as-is — even sparse buckets
// (south_asian: 3, black: 14) just repeat photos rather
// than route to ComfyUI. Repetition is fine; burnt faces
// are not.
pool = gr;
bucket = `gr:${wantGender}/${wantRace}`;
} else if (F.byG[wantGender]?.length) {
pool = F.byG[wantGender];
bucket = `g:${wantGender}`;
}
} else if (wantGender && F.byG[wantGender]?.length) {
pool = F.byG[wantGender];
bucket = `g:${wantGender}`;
} else if (wantRace && F.byR[wantRace]?.length) {
pool = F.byR[wantRace];
bucket = `r:${wantRace}`;
}
// Hash key → pool index. djb2-ish, fits any string. // Hash key → pool index. djb2-ish, fits any string.
let h = 5381; let h = 5381;
for (let i = 0; i < key.length; i++) h = ((h << 5) + h + key.charCodeAt(i)) | 0; for (let i = 0; i < key.length; i++) h = ((h << 5) + h + key.charCodeAt(i)) | 0;
const idx = Math.abs(h) % pool.length; const idx = Math.abs(h) % pool.length;
const pick = pool[idx]; const pick = pool[idx];
// Prefer pre-resized webp thumb (~10KB) over native JPEG
// (~580KB). 60× smaller — without this, a 40-card grid
// overruns Chrome's parallel-connection budget and ~75% of
// tiles never finish decoding.
//
// Cache-Control: 1h public + must-revalidate, NOT immutable.
// We deliberately let the browser re-check after pool retags
// or face-pool refreshes — `immutable` was pinning stale
// photos for 24h after a server-side update.
const thumbName = pick.file.replace(/\.jpg$/, ".webp");
const thumb = Bun.file(`${HEADSHOT_DIR}/_thumbs/${thumbName}`);
if (await thumb.exists()) {
return new Response(thumb, {
headers: {
"Content-Type": "image/webp",
"Cache-Control": "public, max-age=3600, must-revalidate",
"X-Face-Pool-Idx": String(pick.id),
"X-Face-Pool-Gender": pick.gender || "untagged",
"X-Face-Pool-Bucket": bucket,
"X-Face-Pool-Bucket-Size": String(pool.length),
"X-Face-Pool-Variant": "thumb-384",
},
});
}
const file = Bun.file(`${HEADSHOT_DIR}/${pick.file}`); const file = Bun.file(`${HEADSHOT_DIR}/${pick.file}`);
if (!(await file.exists())) { if (!(await file.exists())) {
return new Response("face missing on disk", { status: 404 }); return new Response("face missing on disk", { status: 404 });
@@ -1290,9 +1596,12 @@ async function main() {
return new Response(file, { return new Response(file, {
headers: { headers: {
"Content-Type": "image/jpeg", "Content-Type": "image/jpeg",
"Cache-Control": "public, max-age=86400, immutable", "Cache-Control": "public, max-age=3600, must-revalidate",
"X-Face-Pool-Idx": String(pick.id), "X-Face-Pool-Idx": String(pick.id),
"X-Face-Pool-Gender": pick.gender || "untagged", "X-Face-Pool-Gender": pick.gender || "untagged",
"X-Face-Pool-Bucket": bucket,
"X-Face-Pool-Bucket-Size": String(pool.length),
"X-Face-Pool-Variant": "native-1024",
}, },
}); });
} }
@@ -1772,15 +2081,88 @@ async function main() {
.reduce((s, c) => s + (c.implied_pay_rate - contractBillRate) * hoursPerWeek * weeksAssumed, 0); .reduce((s, c) => s + (c.implied_pay_rate - contractBillRate) * hoursPerWeek * weeksAssumed, 0);
// Shift inference from permit work_type + description. // Shift inference from permit work_type + description.
// Construction defaults to 1st-shift (day). Heavy civil or // Description keywords trump the hash-based assignment;
// facility work sometimes runs 2nd or split-shift. 3rd // for everything else we deterministically distribute
// (overnight) is rare in commercial construction but real // permits across shifts via a hash of the permit id so
// for maintenance / emergency calls. // every shift bucket has real, stable data instead of
// every contract collapsing to 1st.
const descLower = ((p.work_description || "") + " " + (p.work_type || "")).toLowerCase(); const descLower = ((p.work_description || "") + " " + (p.work_type || "")).toLowerCase();
const shifts: string[] = ["1st"]; // default day function hashStr(s: string){
if (/night|overnight|24\s*hr|emergency/.test(descLower)) shifts.push("3rd"); let h=5381;
if (/multi.?shift|round.?the.?clock|double.?shift/.test(descLower)) shifts.push("2nd"); for(let i=0;i<s.length;i++) h=((h<<5)+h+s.charCodeAt(i))|0;
if (/weekend|saturday|sunday/.test(descLower)) shifts.push("4th"); return Math.abs(h);
}
const permitKey = String(p.id || (p.street_number+p.street_name) || p.work_description || "").slice(0,80);
const hh = hashStr(permitKey);
const bucket = hh % 100;
// Realistic split: 50% day, 28% evening, 17% overnight,
// 5% weekend. Construction skews heavily day-shift.
let primary: string =
bucket < 50 ? "1st"
: bucket < 78 ? "2nd"
: bucket < 95 ? "3rd"
: "4th";
const shifts: string[] = [primary];
if (/night|overnight|24\s*hr|emergency/.test(descLower) && !shifts.includes("3rd")) shifts.push("3rd");
if (/multi.?shift|round.?the.?clock|double.?shift/.test(descLower) && !shifts.includes("2nd")) shifts.push("2nd");
if (/weekend|saturday|sunday/.test(descLower) && !shifts.includes("4th")) shifts.push("4th");
// Internal calendar: build a 7-day schedule (today ±3
// days) with a row per (date, shift). This is what the
// front-end's shift-mix preview filters against — real
// dates, real workers/bill, real status (past/active/
// scheduled) tied to the current clock. As permits get
// ingested with explicit start/end dates the backend
// can replace this with the stored schedule.
const SHIFT_HOURS: Record<string, [number, number]> = {
"1st": [6, 14], "2nd": [14, 22], "3rd": [22, 30], "4th": [0, 24], // 4th = weekend
};
function shiftStatus(d: Date, shift: string, ref: Date): "past" | "active" | "scheduled" {
const refDay = ref.toISOString().slice(0,10);
const dDay = d.toISOString().slice(0,10);
if (dDay < refDay) return "past";
if (dDay > refDay) return "scheduled";
// Same day — break by hour vs shift window.
const hr = ref.getHours() + ref.getMinutes()/60;
const [s,e] = SHIFT_HOURS[shift] || [0,24];
if (shift === "4th") {
// Weekend shift: active if today IS weekend, else scheduled.
const isWknd = (ref.getDay()===0 || ref.getDay()===6);
return isWknd ? "active" : "scheduled";
}
if (shift === "3rd") {
// 3rd wraps midnight: active 22:00–06:00.
if (hr >= 22 || hr < 6) return "active";
return "scheduled";
}
if (hr < s) return "scheduled";
if (hr >= e) return "past";
return "active";
}
const refNow = new Date();
const schedule: any[] = [];
for (let off = -3; off <= 3; off++) {
const d = new Date(refNow.getTime() + off * 86400e3);
const isWknd = (d.getDay()===0 || d.getDay()===6);
const dateStr = d.toISOString().slice(0,10);
for (const sh of shifts) {
// Weekend permits use 4th shift only; weekday work
// uses its primary shift(s) and skips 4th.
if (isWknd && sh !== "4th") continue;
if (!isWknd && sh === "4th") continue;
// Workers per shift: full count on primary, half on
// secondary so the bill demand differs visibly.
const isPrimary = (sh === primary);
const wForShift = isPrimary ? count : Math.max(1, Math.floor(count/2));
schedule.push({
date: dateStr,
shift: sh,
workers_needed: wForShift,
bill_rate: contractBillRate,
status: shiftStatus(d, sh, refNow),
});
}
}
contracts.push({ contracts.push({
permit: { permit: {
@@ -1835,6 +2217,7 @@ async function main() {
over_bill_pool_margin_at_risk: Math.round(overBillPoolMargin), over_bill_pool_margin_at_risk: Math.round(overBillPoolMargin),
}, },
shifts_needed: shifts, shifts_needed: shifts,
schedule,
}); });
} }
mcp-server/observer.ts
+10 -8
View File
@@ -146,15 +146,16 @@ async function persistOp(op: ObservedOp) {
// ─── LLM Team escalation (code_review mode) ─── // ─── LLM Team escalation (code_review mode) ───
// //
// When recent failures on a single sig_hash cross a threshold the // When recent failures on a single sig_hash cross a threshold the
// local qwen2.5 analysis is probably insufficient. J's 2026-04-24 // local-model analysis is probably insufficient. J's 2026-04-24
// direction: "the observer would trigger to give more context" — // direction: "the observer would trigger to give more context" —
// route failure clusters to LLM Team's specialized code_review mode // route failure clusters to LLM Team's specialized code_review mode
// (via /api/run) so richer structured signal lands in the KB for // (via /api/run) so richer structured signal lands in the KB for
// scrum + auditor + playbook memory to consume next pass. // scrum + auditor + playbook memory to consume next pass.
// //
// Non-destructive: runs in parallel to the existing qwen2.5 analysis, // Non-destructive: runs in parallel to the existing local diagnose
// never replaces it. Writes to data/_kb/observer_escalations.jsonl // call (qwen3.5:latest after the 2026-04-30 bump), never replaces
// as a dedicated audit surface. // it. Writes to data/_kb/observer_escalations.jsonl as a dedicated
// audit surface.
const LLM_TEAM = process.env.LH_LLM_TEAM_URL ?? "http://localhost:5000"; const LLM_TEAM = process.env.LH_LLM_TEAM_URL ?? "http://localhost:5000";
const LLM_TEAM_ESCALATIONS = "/home/profit/lakehouse/data/_kb/observer_escalations.jsonl"; const LLM_TEAM_ESCALATIONS = "/home/profit/lakehouse/data/_kb/observer_escalations.jsonl";
@@ -542,7 +543,7 @@ async function analyzeErrors() {
if (failures.length === 0) return; if (failures.length === 0) return;
// NEW 2026-04-24: escalate recurring sig_hash clusters to LLM Team // NEW 2026-04-24: escalate recurring sig_hash clusters to LLM Team
// code_review mode. Runs in parallel to the local qwen2.5 analysis // code_review mode. Runs in parallel to the local diagnose call
// below — non-blocking, richer downstream signal for scrum/auditor. // below — non-blocking, richer downstream signal for scrum/auditor.
maybeEscalate(failures).catch(() => {}); maybeEscalate(failures).catch(() => {});
@@ -552,13 +553,14 @@ async function analyzeErrors() {
// Ask local model to diagnose. Phase 44 migration (2026-04-27): // Ask local model to diagnose. Phase 44 migration (2026-04-27):
// /v1/chat instead of legacy /ai/generate so /v1/usage tracks the // /v1/chat instead of legacy /ai/generate so /v1/usage tracks the
// call + Langfuse traces it. Same upstream model (qwen2.5 local). // call + Langfuse traces it. 2026-04-30 model bump: qwen2.5 →
// qwen3.5:latest to match the small-model-pipeline local-tier default.
try { try {
const resp = await fetch(`${LAKEHOUSE}/v1/chat`, { const resp = await fetch(`${LAKEHOUSE}/v1/chat`, {
method: "POST", method: "POST",
headers: { "Content-Type": "application/json" }, headers: { "Content-Type": "application/json" },
body: JSON.stringify({ body: JSON.stringify({
model: "qwen2.5", model: "qwen3.5:latest",
provider: "ollama", provider: "ollama",
messages: [{ messages: [{
role: "user", role: "user",
@@ -769,7 +771,7 @@ async function tailOverseerCorrections(): Promise<number> {
try { row = JSON.parse(line); } catch { continue; } try { row = JSON.parse(line); } catch { continue; }
const op: ObservedOp = { const op: ObservedOp = {
timestamp: row.created_at ?? new Date().toISOString(), timestamp: row.created_at ?? new Date().toISOString(),
endpoint: `overseer:${row.model ?? "gpt-oss:120b"}`, endpoint: `overseer:${row.model ?? "claude-opus-4-7"}`,
input_summary: `${row.task_class ?? "?"}: ${row.reason ?? "escalation"}`, input_summary: `${row.task_class ?? "?"}: ${row.reason ?? "escalation"}`,
// Correction itself is neither success nor failure — it's a // Correction itself is neither success nor failure — it's a
// mitigation attempt. We mark success=true so analyzeErrors // mitigation attempt. We mark success=true so analyzeErrors
mcp-server/role_scenes.ts
+92
View File
@@ -0,0 +1,92 @@
// Server-side mirror of search.html's ROLE_BANDS regex table.
// Each band carries a *visual scene* — clothing + immediate backdrop —
// so ComfyUI produces role-coherent headshots instead of interchangeable
// studio portraits. The front-end sends the raw role string in the
// query (?role=Forklift%20Operator); the server resolves it to a band
// and looks up the scene here.
export type RoleBand =
| "warehouse"
| "production"
| "trades"
| "driver"
| "lead";
export interface SceneDef {
band: RoleBand;
// Free-form clause inserted into the diffusion prompt AFTER
// "[age]-year-old [race] [gender] [role], ". Should describe what
// they're wearing and what is immediately behind them. Keep under
// ~25 words — SDXL Turbo loses focus on longer prompts and starts
// hallucinating cartoon hands.
scene: string;
}
const RE_BANDS: { re: RegExp; band: RoleBand }[] = [
{ re: /forklift|warehouse|associate|material\s*handler|loader|loading|packag|shipping|logistics|inventory|sanitation|janit/i, band: "warehouse" },
{ re: /production|assembl|quality/i, band: "production" },
{ re: /welder|weld|electric|maint(enance)?\s*tech|cnc|machine\s*op|hvac|plumb|carpenter|mason|tool\s*&\s*die/i, band: "trades" },
{ re: /driver|truck|haul|cdl/i, band: "driver" },
{ re: /line\s*lead|supervisor|foreman|coordinator|lead\b/i, band: "lead" },
];
export function roleBand(role: string): RoleBand {
const r = (role || "").trim();
if (!r) return "warehouse";
for (const b of RE_BANDS) if (b.re.test(r)) return b.band;
return "warehouse";
}
// TODO J — refine these. Each `scene` string lands directly in the
// diffusion prompt. Tone target: a coordinator glances at the card
// and recognizes the role from the photo before reading the role pill.
//
// Things that work well in SDXL Turbo at 8 steps:
// - One concrete clothing item ("high-visibility yellow vest")
// - One concrete prop ("hard hat hanging from belt", "tablet in hand")
// - One blurred background element ("warehouse pallet aisle behind",
// "factory machinery softly out of focus")
// - Avoid: text/logos (rendered as scribble), specific brands, hands
// holding tools (often distorts), full-body language ("standing",
// "leaning") — model is trained on portrait crops.
//
// Each scene now bakes "monochrome black and white photography" into
// the prompt so the model produces native B&W output rather than us
// applying CSS grayscale post-hoc. SDXL Turbo handles B&W natively
// with strong tonal range — better than desaturating a color render.
export const SCENES: Record<RoleBand, SceneDef> = {
warehouse: {
band: "warehouse",
scene: "wearing a high-visibility safety vest over a t-shirt, hard hat visible, blurred warehouse pallet aisle behind, soft natural light, monochrome black and white photography, fine film grain, documentary portrait style",
},
production: {
band: "production",
scene: "wearing a work shirt with safety glasses on forehead, blurred factory machinery softly out of focus behind, fluorescent overhead lighting, monochrome black and white photography, fine film grain, documentary portrait style",
},
trades: {
band: "trades",
scene: "wearing a heavy-duty work shirt with rolled sleeves, blurred workshop tool wall behind, focused tungsten lighting, monochrome black and white photography, fine film grain, documentary portrait style",
},
driver: {
band: "driver",
scene: "wearing a polo shirt, lanyard with ID badge visible, blurred truck cab or loading dock behind, daylight, monochrome black and white photography, fine film grain, documentary portrait style",
},
lead: {
band: "lead",
scene: "wearing a button-down shirt, tablet held casually at chest level, blurred warehouse floor in soft focus behind, professional lighting, monochrome black and white photography, fine film grain, documentary portrait style",
},
};
// v2 — baked B&W + 1024×1024 render canvas (4× pixels of v1). Larger
// source means downsampling to a 40px avatar packs more detail per
// displayed pixel, hiding the diffusion-y micro-textures that read as
// "AI generated" at small sizes. Server route reads pool from
// data/headshots_role_pool/{SCENES_VERSION}/... so v1 stays available
// for rollback / A-B comparison.
export const SCENES_VERSION = "v2";
// Default render dimensions used by both the on-demand /headshots/
// generate/:key route and the offline render_role_pool.py script. v1
// used 512²; v2 doubles to 1024² (linear 2× = 4× pixels = ~3× GPU
// time on SDXL Turbo).
export const FACE_RENDER_DIM = 1024;
mcp-server/search.html
+1308 -161
View File
File diff suppressed because it is too large Load Diff
mcp-server/tif_polygons.ts
+178
View File
@@ -0,0 +1,178 @@
// TIF (Tax Increment Financing) district point-in-polygon lookup.
// Given a property's lat/long, returns which Chicago TIF district (if
// any) contains it. TIF districts are public-subsidy zones — a property
// inside one is receiving city tax-increment funding for its build.
// Strong "this project has financial backing" signal for the Project Index.
//
// Data: data/_entity_cache/tif_districts.geojson (Chicago Open Data
// dataset eejr-xtfb, 100 active districts, 3.2MB). Refresh by re-running
// `curl ... eejr-xtfb.geojson > tif_districts.geojson` — districts
// change rarely (only when city council approves new ones or repeals).
//
// Algorithm: classic ray-casting. For each MultiPolygon's outer ring,
// count edge crossings of an east-going horizontal ray from the point.
// Odd crossings = inside. Holes (inner rings) flip the parity. Library-
// free; correct for arbitrary polygons including the irregular Chicago
// shapes which often have many small detours.
import { readFile } from "node:fs/promises";
import { existsSync } from "node:fs";
import { join } from "node:path";
const TIF_GEOJSON = join("/home/profit/lakehouse/data/_entity_cache", "tif_districts.geojson");
type LngLat = [number, number]; // GeoJSON convention: [longitude, latitude]
type Ring = LngLat[];
type Polygon = Ring[]; // outer ring + optional inner rings (holes)
type MultiPolygon = Polygon[];
type TifFeature = {
name: string;
trim_name?: string;
ref?: string;
approval_date?: string;
expiration?: string;
type?: string; // T-1xx etc.
comm_area?: string;
wards?: string;
// Bounding box for quick reject
bbox: { minLon: number; minLat: number; maxLon: number; maxLat: number };
geometry: MultiPolygon;
};
let tifIdx: TifFeature[] | null = null;
function bboxOfMultiPolygon(mp: MultiPolygon): TifFeature["bbox"] {
let minLon = Infinity, minLat = Infinity, maxLon = -Infinity, maxLat = -Infinity;
for (const poly of mp) {
for (const ring of poly) {
for (const [lon, lat] of ring) {
if (lon < minLon) minLon = lon;
if (lat < minLat) minLat = lat;
if (lon > maxLon) maxLon = lon;
if (lat > maxLat) maxLat = lat;
}
}
}
return { minLon, minLat, maxLon, maxLat };
}
async function ensureLoaded(): Promise<TifFeature[]> {
if (tifIdx) return tifIdx;
if (!existsSync(TIF_GEOJSON)) {
tifIdx = [];
return tifIdx;
}
try {
const raw = JSON.parse(await readFile(TIF_GEOJSON, "utf-8"));
const out: TifFeature[] = [];
for (const f of raw.features || []) {
const geom = f.geometry;
if (!geom) continue;
// Normalize Polygon → MultiPolygon for uniform iteration
let mp: MultiPolygon;
if (geom.type === "MultiPolygon") {
mp = geom.coordinates;
} else if (geom.type === "Polygon") {
mp = [geom.coordinates];
} else {
continue;
}
const props = f.properties || {};
out.push({
name: props.name || "Unknown TIF",
trim_name: props.name_trim,
ref: props.ref,
approval_date: props.approval_d,
expiration: props.expiration,
type: props.type,
comm_area: props.comm_area,
wards: props.wards,
bbox: bboxOfMultiPolygon(mp),
geometry: mp,
});
}
tifIdx = out;
return tifIdx;
} catch (e) {
console.warn("[tif] load failed:", (e as Error).message);
tifIdx = [];
return tifIdx;
}
}
// Ray-casting point-in-polygon (single ring). Returns true if (lon, lat)
// is strictly inside the ring. Edge cases (point exactly on a vertex)
// resolve by half-open interval convention; for our use case (Chicago
// boundary precision is ~1m, sites are point queries) this is fine.
function pointInRing(lon: number, lat: number, ring: Ring): boolean {
let inside = false;
const n = ring.length;
for (let i = 0, j = n - 1; i < n; j = i++) {
const [xi, yi] = ring[i];
const [xj, yj] = ring[j];
const intersect =
yi > lat !== yj > lat &&
lon < ((xj - xi) * (lat - yi)) / (yj - yi + 0) + xi;
if (intersect) inside = !inside;
}
return inside;
}
// Polygon = outer ring + holes. Inside outer AND not inside any hole.
function pointInPolygon(lon: number, lat: number, polygon: Polygon): boolean {
if (polygon.length === 0) return false;
if (!pointInRing(lon, lat, polygon[0])) return false;
for (let i = 1; i < polygon.length; i++) {
if (pointInRing(lon, lat, polygon[i])) return false;
}
return true;
}
export type TifMatch = {
name: string;
ref?: string;
approval_date?: string;
expiration?: string;
comm_area?: string;
wards?: string;
};
export async function findTifDistrict(
longitude: number | string | undefined,
latitude: number | string | undefined,
): Promise<TifMatch | null> {
const lon = typeof longitude === "string" ? parseFloat(longitude) : longitude;
const lat = typeof latitude === "string" ? parseFloat(latitude) : latitude;
if (!lon || !lat || isNaN(lon) || isNaN(lat)) return null;
const idx = await ensureLoaded();
if (idx.length === 0) return null;
for (const f of idx) {
// Bbox reject — cheap O(1) skip for the 99% of districts that
// can't possibly contain the point.
const b = f.bbox;
if (lon < b.minLon || lon > b.maxLon || lat < b.minLat || lat > b.maxLat) continue;
// Full point-in-polygon for any polygon in this MultiPolygon
for (const poly of f.geometry) {
if (pointInPolygon(lon, lat, poly)) {
return {
name: f.name,
ref: f.ref,
approval_date: f.approval_date,
expiration: f.expiration,
comm_area: f.comm_area,
wards: f.wards,
};
}
}
}
return null;
}
export async function getTifIndexStats(): Promise<{
total: number;
loaded: boolean;
}> {
const idx = await ensureLoaded();
return { total: idx.length, loaded: idx.length > 0 };
}
reports/distillation/phase6-acceptance-report.md
+4 -4
View File
@@ -1,6 +1,6 @@
# Phase 6 — Acceptance Gate Report # Phase 6 — Acceptance Gate Report
**Run:** 2026-04-27T04:54:32.225Z **Run:** 2026-04-27T15:43:37.943Z
**Fixture:** `tests/fixtures/distillation/acceptance/` **Fixture:** `tests/fixtures/distillation/acceptance/`
**Temp root:** `/tmp/distillation_phase6_acceptance` **Temp root:** `/tmp/distillation_phase6_acceptance`
**Pipeline run_ids:** `acceptance-run-1-stable` (first) + `acceptance-run-2-stable` (second / hash reproducibility) **Pipeline run_ids:** `acceptance-run-1-stable` (first) + `acceptance-run-2-stable` (second / hash reproducibility)
@@ -40,13 +40,13 @@
| 19 | scratchpad/tree-split case: fixture row materialized into evidence | found | found | ✓ | | 19 | scratchpad/tree-split case: fixture row materialized into evidence | found | found | ✓ |
| 20 | PRD drift case: fixture row materialized | found | found | ✓ | | 20 | PRD drift case: fixture row materialized | found | found | ✓ |
| 21 | hash reproducibility: per-stage output_hash identical across runs | 0 mismatches | all match | ✓ | | 21 | hash reproducibility: per-stage output_hash identical across runs | 0 mismatches | all match | ✓ |
| 22 | hash reproducibility: run_hash identical | 3ea12b160ee9099a... | 3ea12b160ee9099a... | ✓ | | 22 | hash reproducibility: run_hash identical | 8dfdacee62380ec2... | 8dfdacee62380ec2... | ✓ |
## Hash reproducibility detail ## Hash reproducibility detail
run 1 run_hash: `3ea12b160ee9099a3c52fe6e7fffd3076de7920d2704d24c789260d63cb1a5a2` run 1 run_hash: `8dfdacee62380ec20b7420d8f8bad3c395822da6eb0b41eeecd356e88fe20bf0`
run 2 run_hash: `3ea12b160ee9099a3c52fe6e7fffd3076de7920d2704d24c789260d63cb1a5a2` run 2 run_hash: `8dfdacee62380ec20b7420d8f8bad3c395822da6eb0b41eeecd356e88fe20bf0`
**Bit-for-bit identical.** Two runs of the entire pipeline on the same fixture with the same `recorded_at` produce the same outputs. Distillation is deterministic. **Bit-for-bit identical.** Two runs of the entire pipeline on the same fixture with the same `recorded_at` produce the same outputs. Distillation is deterministic.
reports/distillation/phase8-full-audit-report.md
+8 -8
View File
@@ -1,8 +1,8 @@
# Phase 8 — Full System Audit Report # Phase 8 — Full System Audit Report
**Run:** 2026-04-27T04:54:32.283Z **Run:** 2026-04-27T15:43:38.021Z
**Git commit:** 73f242e3e41c2aa36b35fe9de54742b248915cb5 **Git commit:** ca7375ea2b178159a0c61bbf62788a2ffa2390e9
**Baseline:** 2026-04-27T04:53:45.796Z (5bdd159966e6) **Baseline:** 2026-04-27T10:31:44.043Z (d11632a6fae6)
## Result: **PASS** ✓ ## Result: **PASS** ✓
@@ -26,7 +26,7 @@
| 1 | P0 | recon doc exists | Y | docs/recon/local-distillation-recon.md present | present | ✓ | | 1 | P0 | recon doc exists | Y | docs/recon/local-distillation-recon.md present | present | ✓ |
| 2 | P0 | tier-1 source streams present | — | all 4 tier-1 jsonls on disk | all present | ✓ | | 2 | P0 | tier-1 source streams present | — | all 4 tier-1 jsonls on disk | all present | ✓ |
| 3 | P1 | schema validators pass on fixtures | Y | ≥40 tests, 0 fail | 51 pass, 0 fail | ✓ | | 3 | P1 | schema validators pass on fixtures | Y | ≥40 tests, 0 fail | 51 pass, 0 fail | ✓ |
| 4 | P2 | materializer dry-run completes | Y | >=1 row from each tier-1 source | 1073 read · 16 written · 2 skipped | ✓ | | 4 | P2 | materializer dry-run completes | Y | >=1 row from each tier-1 source | 1139 read · 82 written · 2 skipped | ✓ |
| 5 | P2 | tier-1 sources each materialize ≥1 row | — | 4/4: distilled_facts, scrum_reviews, audit_facts, mode_experiments | 1/4 hit (mode_experiments) | ✓ | | 5 | P2 | tier-1 sources each materialize ≥1 row | — | 4/4: distilled_facts, scrum_reviews, audit_facts, mode_experiments | 1/4 hit (mode_experiments) | ✓ |
| 6 | P3 | on-disk scored-runs distribution non-empty | Y | >=1 accepted | acc=386 part=132 rej=57 hum=480 | ✓ | | 6 | P3 | on-disk scored-runs distribution non-empty | Y | >=1 accepted | acc=386 part=132 rej=57 hum=480 | ✓ |
| 7 | P3 | scored-runs distribution sums positive | — | >0 total | 1055 total | ✓ | | 7 | P3 | scored-runs distribution sums positive | — | >0 total | 1055 total | ✓ |
@@ -38,19 +38,19 @@
| 13 | P5 | latest run (3fa51d66-784c-4c7d-843d-6c48328a608c) has all 5 stage receipts | Y | collect,score,export-rag,export-sft,export-preference | all present | ✓ | | 13 | P5 | latest run (3fa51d66-784c-4c7d-843d-6c48328a608c) has all 5 stage receipts | Y | collect,score,export-rag,export-sft,export-preference | all present | ✓ |
| 14 | P5 | every stage receipt validates against schema | Y | 0 invalid | 0 invalid | ✓ | | 14 | P5 | every stage receipt validates against schema | Y | 0 invalid | 0 invalid | ✓ |
| 15 | P5 | RunSummary validates | Y | valid | valid | ✓ | | 15 | P5 | RunSummary validates | Y | valid | valid | ✓ |
| 16 | P5 | summary.git_commit is 40-char hex | — | match | 68b6697bcb38... (HEAD: 73f242e3e41c...) | ✓ | | 16 | P5 | summary.git_commit is 40-char hex | — | match | 68b6697bcb38... (HEAD: ca7375ea2b17...) | ✓ |
| 17 | P5 | run_hash is sha256 | Y | /^[0-9a-f]{64}$/ | 2336b96c3638982d... | ✓ | | 17 | P5 | run_hash is sha256 | Y | /^[0-9a-f]{64}$/ | 2336b96c3638982d... | ✓ |
| 18 | P6 | acceptance gate passes 22/22 invariants on fixture | Y | PASS — 22/22 | 22/22 (exit=0) | ✓ | | 18 | P6 | acceptance gate passes 22/22 invariants on fixture | Y | PASS — 22/22 | 22/22 (exit=0) | ✓ |
| 19 | P7 | replay validation passes on 3/3 dry-run sample tasks | Y | 3/3 | 3/3 | ✓ | | 19 | P7 | replay validation passes on 3/3 dry-run sample tasks | Y | 3/3 | 3/3 | ✓ |
| 20 | P7 | replay retrieval surfaces ≥1 playbook on each task (when corpus present) | — | ≥1 task with retrieval | 3/3 | ✓ | | 20 | P7 | replay retrieval surfaces ≥1 playbook on each task (when corpus present) | — | ≥1 task with retrieval | 3/3 | ✓ |
| 21 | P7 | escalation loop guard: no path > 2 models | Y | 0 loops | 0 | ✓ | | 21 | P7 | escalation loop guard: no path > 2 models | Y | 0 loops | 0 | ✓ |
| 22 | P7 | replay_runs.jsonl populated by audit run | — | exists with ≥3 rows added | 21 rows total | ✓ | | 22 | P7 | replay_runs.jsonl populated by audit run | — | exists with ≥3 rows added | 27 rows total | ✓ |
## Drift vs prior baseline ## Drift vs prior baseline
| Metric | Baseline | Current | Δ% | Flag | | Metric | Baseline | Current | Δ% | Flag |
|---|---|---|---|---| |---|---|---|---|---|
| p2_evidence_rows | 15 | 16 | 7% | ok | | p2_evidence_rows | 25 | 82 | 228% | warn |
| p2_evidence_skips | 2 | 2 | 0% | ok | | p2_evidence_skips | 2 | 2 | 0% | ok |
| p3_accepted | 386 | 386 | 0% | ok | | p3_accepted | 386 | 386 | 0% | ok |
| p3_partial | 132 | 132 | 0% | ok | | p3_partial | 132 | 132 | 0% | ok |
@@ -61,7 +61,7 @@
| p4_pref_pairs | 83 | 83 | 0% | ok | | p4_pref_pairs | 83 | 83 | 0% | ok |
| p4_total_quarantined | 1325 | 1325 | 0% | ok | | p4_total_quarantined | 1325 | 1325 | 0% | ok |
All metrics within 20% of baseline — pipeline stable across runs. **1 metric(s) drifted >20% from baseline.** Investigate before treating outputs as stable.
## System health status ## System health status
scripts/serve_imagegen.py
+52 -7
View File
@@ -29,8 +29,14 @@ CACHE_DIR.mkdir(parents=True, exist_ok=True)
WORKFLOW_PATH = "/opt/ComfyUI/workflows/editorial_hero.json" WORKFLOW_PATH = "/opt/ComfyUI/workflows/editorial_hero.json"
def _cache_key(prompt, width, height, steps): def _cache_key(prompt, width, height, steps, seed=None):
return hashlib.sha256(f"{prompt}|{width}|{height}|{steps}".encode()).hexdigest()[:24] # Include seed so callers can vary outputs deterministically without
# the proxy collapsing to a single cached image. None == legacy
# (omitted from the key for backward compatibility).
bits = f"{prompt}|{width}|{height}|{steps}"
if seed is not None:
bits += f"|{seed}"
return hashlib.sha256(bits.encode()).hexdigest()[:24]
def _cache_get(key): def _cache_get(key):
fp = CACHE_DIR / f"{key}.webp" fp = CACHE_DIR / f"{key}.webp"
@@ -40,8 +46,15 @@ def _cache_put(key, img_bytes):
(CACHE_DIR / f"{key}.webp").write_bytes(img_bytes) (CACHE_DIR / f"{key}.webp").write_bytes(img_bytes)
def _comfyui_generate(prompt, width=1024, height=512, steps=8, seed=None): def _comfyui_generate(prompt, width=1024, height=512, steps=8, seed=None,
"""Submit workflow to ComfyUI and wait for result.""" negative_prompt=None, cfg=None, sampler=None, scheduler=None):
"""Submit workflow to ComfyUI and wait for result.
Optional overrides — when provided, replace the workflow's defaults.
The workflow template at editorial_hero.json was tuned for product
hero shots with a "no humans" negative prompt; portrait callers MUST
pass `negative_prompt` to avoid the model fighting them on faces.
"""
# Load workflow template # Load workflow template
with open(WORKFLOW_PATH) as f: with open(WORKFLOW_PATH) as f:
workflow = json.load(f) workflow = json.load(f)
@@ -51,9 +64,21 @@ def _comfyui_generate(prompt, width=1024, height=512, steps=8, seed=None):
seed = random.randint(0, 2**32) seed = random.randint(0, 2**32)
workflow["3"]["inputs"]["seed"] = seed workflow["3"]["inputs"]["seed"] = seed
workflow["3"]["inputs"]["steps"] = steps workflow["3"]["inputs"]["steps"] = steps
if cfg is not None:
workflow["3"]["inputs"]["cfg"] = cfg
if sampler:
workflow["3"]["inputs"]["sampler_name"] = sampler
if scheduler:
workflow["3"]["inputs"]["scheduler"] = scheduler
workflow["5"]["inputs"]["width"] = width workflow["5"]["inputs"]["width"] = width
workflow["5"]["inputs"]["height"] = height workflow["5"]["inputs"]["height"] = height
workflow["6"]["inputs"]["text"] = prompt workflow["6"]["inputs"]["text"] = prompt
# Node 7 is the negative-prompt CLIPTextEncode. The default is tuned
# for product hero shots and contains "human, person, face, hand,
# fingers, realistic photo of people" — actively sabotaging any
# portrait render. Always overwrite when negative_prompt is given.
if negative_prompt is not None:
workflow["7"]["inputs"]["text"] = negative_prompt
# Submit to ComfyUI # Submit to ComfyUI
payload = json.dumps({"prompt": workflow}).encode() payload = json.dumps({"prompt": workflow}).encode()
@@ -177,9 +202,20 @@ class ImageHandler(BaseHTTPRequestHandler):
height = min(max(int(body.get("height", 720)), 256), 1080) height = min(max(int(body.get("height", 720)), 256), 1080)
steps = min(max(int(body.get("steps", 50)), 1), 80) steps = min(max(int(body.get("steps", 50)), 1), 80)
seed = body.get("seed") seed = body.get("seed")
# Portrait-friendly overrides — None means "use workflow default".
# negative_prompt MUST be passed by portrait callers to avoid
# the workflow's "no humans" baked-in negative.
negative_prompt = body.get("negative_prompt")
cfg = body.get("cfg")
sampler = body.get("sampler")
scheduler = body.get("scheduler")
# Cache check # Cache check — seed + negative + cfg are part of the key so per-
key = _cache_key(prompt, width, height, steps) # worker / per-config requests don't collapse to one cached image.
key = _cache_key(
f"{prompt}||neg={negative_prompt or ''}||cfg={cfg or ''}",
width, height, steps, seed,
)
cached = _cache_get(key) cached = _cache_get(key)
if cached: if cached:
self._json(200, {"image": cached, "format": "webp", "width": width, "height": height, self._json(200, {"image": cached, "format": "webp", "width": width, "height": height,
@@ -192,7 +228,11 @@ class ImageHandler(BaseHTTPRequestHandler):
try: try:
comfy_check = urllib.request.urlopen(f"{COMFYUI_URL}/system_stats", timeout=3) comfy_check = urllib.request.urlopen(f"{COMFYUI_URL}/system_stats", timeout=3)
if comfy_check.status == 200: if comfy_check.status == 200:
img_bytes, seed = _comfyui_generate(prompt, width, height, steps, seed) img_bytes, seed = _comfyui_generate(
prompt, width, height, steps, seed,
negative_prompt=negative_prompt, cfg=cfg,
sampler=sampler, scheduler=scheduler,
)
backend = "comfyui" backend = "comfyui"
except: except:
pass pass
@@ -210,6 +250,11 @@ class ImageHandler(BaseHTTPRequestHandler):
elapsed_ms = int((time.time() - t0) * 1000) elapsed_ms = int((time.time() - t0) * 1000)
img_b64 = base64.b64encode(img_bytes).decode() img_b64 = base64.b64encode(img_bytes).decode()
# Recompute key with the actual seed used (when caller passed
# None, _comfyui_generate picks a random one and we want the
# cache to reflect that so re-requests with the same returned
# seed hit the disk).
key = _cache_key(prompt, width, height, steps, seed)
_cache_put(key, img_bytes) _cache_put(key, img_bytes)
self._json(200, { self._json(200, {
scripts/staffing/fetch_face_pool.py
+78 -14
View File
@@ -53,20 +53,30 @@ def fetch_one(idx: int, out_dir: str) -> tuple[int, str, bool, str | None]:
def maybe_tag_gender(records: list[dict], out_dir: str) -> dict[str, int]: def maybe_tag_gender(records: list[dict], out_dir: str) -> dict[str, int]:
"""If deepface is installed, label each record with gender. Returns """If deepface is installed, label records that don't already have a
a count summary; mutates records in place. On import error, returns gender. Returns a count summary; mutates records in place.
None and tags every record as unknown."""
Preservation contract: never overwrites prior `gender` (or any other
tag — race/age/excluded — set by tag_face_pool.py). On deepface
import failure, leaves existing tags alone instead of resetting them
to None. The previous behavior wiped 952 hand-classified rows when
fetch_face_pool was re-run from a Python without deepface installed."""
try: try:
from deepface import DeepFace # type: ignore from deepface import DeepFace # type: ignore
except Exception as e: except Exception as e:
print(f" (deepface unavailable: {e}) — pool will mix naturally") print(f" (deepface unavailable: {e}) — leaving existing tags untouched")
for r in records: for r in records:
r["gender"] = None r.setdefault("gender", None)
return {"unknown": len(records)} already = sum(1 for r in records if r.get("gender") in ("man", "woman"))
return {"preserved_tagged": already, "untagged": len(records) - already}
print(" tagging gender via deepface (CPU; ~0.5-1s per face)…") todo = [r for r in records if r.get("gender") not in ("man", "woman")]
if not todo:
print(" every record already has gender — nothing to tag.")
return {"preserved_tagged": len(records)}
print(f" tagging gender via deepface ({len(todo)} of {len(records)} records, CPU; ~0.5-1s per face)…")
counts: dict[str, int] = {} counts: dict[str, int] = {}
for i, r in enumerate(records): for i, r in enumerate(todo):
full = os.path.join(out_dir, r["file"]) full = os.path.join(out_dir, r["file"])
try: try:
ana = DeepFace.analyze( ana = DeepFace.analyze(
@@ -88,7 +98,7 @@ def maybe_tag_gender(records: list[dict], out_dir: str) -> dict[str, int]:
r["gender_error"] = f"{type(e).__name__}: {e}" r["gender_error"] = f"{type(e).__name__}: {e}"
counts[r["gender"] or "unknown"] = counts.get(r["gender"] or "unknown", 0) + 1 counts[r["gender"] or "unknown"] = counts.get(r["gender"] or "unknown", 0) + 1
if (i + 1) % 25 == 0: if (i + 1) % 25 == 0:
print(f" [{i+1}/{len(records)}] {counts}") print(f" [{i+1}/{len(todo)}] {counts}")
return counts return counts
@@ -101,11 +111,47 @@ def main():
) )
p.add_argument("--concurrency", type=int, default=3, help="parallel fetches (be polite)") p.add_argument("--concurrency", type=int, default=3, help="parallel fetches (be polite)")
p.add_argument("--no-gender", action="store_true", help="skip deepface gender tagging") p.add_argument("--no-gender", action="store_true", help="skip deepface gender tagging")
p.add_argument("--shrink", action="store_true",
help="allow --count to drop manifest entries with id >= count. Default: preserve them.")
args = p.parse_args() args = p.parse_args()
out = os.path.realpath(args.out) out = os.path.realpath(args.out)
os.makedirs(out, exist_ok=True) os.makedirs(out, exist_ok=True)
# Load any existing manifest into a by-id dict so prior tags
# (gender / race / age / excluded) survive the rewrite. Also
# naturally dedupes — if the file accidentally has duplicate
# lines for the same id (this is how we ended up with a 2497-
# row manifest backing a 1000-face pool), the last one wins.
manifest = os.path.join(out, "manifest.jsonl")
existing: dict[int, dict] = {}
if os.path.exists(manifest):
dup_count = 0
with open(manifest) as f:
for line in f:
line = line.strip()
if not line:
continue
try:
row = json.loads(line)
except json.JSONDecodeError:
continue
rid = row.get("id")
if not isinstance(rid, int):
continue
if rid in existing:
dup_count += 1
existing[rid] = row
print(f"Loaded existing manifest: {len(existing)} unique ids ({dup_count} duplicate lines collapsed)")
max_existing = max(existing.keys()) if existing else -1
if max_existing >= args.count and not args.shrink:
print(
f"\nERROR: --count={args.count} would drop {sum(1 for k in existing if k >= args.count)} "
f"manifest entries (max existing id = {max_existing}). Pass --shrink to allow.\n",
file=sys.stderr,
)
sys.exit(2)
print(f"Fetching {args.count} faces → {out}") print(f"Fetching {args.count} faces → {out}")
print(f"Source: {URL} (synthetic StyleGAN — no real people)") print(f"Source: {URL} (synthetic StyleGAN — no real people)")
@@ -115,12 +161,16 @@ def main():
futs = {ex.submit(fetch_one, i, out): i for i in range(args.count)} futs = {ex.submit(fetch_one, i, out): i for i in range(args.count)}
for done, fut in enumerate(as_completed(futs), 1): for done, fut in enumerate(as_completed(futs), 1):
idx, fname, cached, err = fut.result() idx, fname, cached, err = fut.result()
results[idx] = { # Start from prior manifest row (preserves gender/race/age/excluded)
# and overlay only the fields fetch_one is responsible for.
base = dict(existing.get(idx, {}))
base.update({
"id": idx, "id": idx,
"file": fname, "file": fname,
"cached": cached, "cached": cached,
"error": err, "error": err,
} })
results[idx] = base
if done % 25 == 0 or done == args.count: if done % 25 == 0 or done == args.count:
ok = sum(1 for r in results if r and not r.get("error")) ok = sum(1 for r in results if r and not r.get("error"))
print(f" [{done}/{args.count}] {ok} ok ({time.time()-t0:.1f}s)") print(f" [{done}/{args.count}] {ok} ok ({time.time()-t0:.1f}s)")
@@ -128,6 +178,9 @@ def main():
# Drop slots that errored or are still None (shouldn't happen) # Drop slots that errored or are still None (shouldn't happen)
records = [r for r in results if r and not r.get("error")] records = [r for r in results if r and not r.get("error")]
print(f"\nPool ready: {len(records)} faces, {sum(1 for r in records if r['cached'])} from cache") print(f"\nPool ready: {len(records)} faces, {sum(1 for r in records if r['cached'])} from cache")
preserved_tags = sum(1 for r in records if r.get("gender") in ("man", "woman"))
if preserved_tags:
print(f"Preserved {preserved_tags} prior gender tags (and any race/age/excluded fields).")
if not args.no_gender and records: if not args.no_gender and records:
print("\nGender-tagging pass:") print("\nGender-tagging pass:")
@@ -135,17 +188,28 @@ def main():
print(f" distribution: {summary}") print(f" distribution: {summary}")
else: else:
for r in records: for r in records:
r["gender"] = None r.setdefault("gender", None)
# If --shrink was NOT used and somehow id >= count rows are still in
# `existing` (which can only happen if the early gate was bypassed),
# carry them forward so we don't quietly drop them.
if not args.shrink:
for rid, row in existing.items():
if rid >= args.count and rid not in {r["id"] for r in records}:
records.append(row)
records.sort(key=lambda r: r.get("id", 0))
# Strip transient flags before persisting # Strip transient flags before persisting
for r in records: for r in records:
r.pop("cached", None) r.pop("cached", None)
r.pop("error", None) r.pop("error", None)
manifest = os.path.join(out, "manifest.jsonl") # Atomic write — if a re-run is interrupted, manifest stays intact.
with open(manifest, "w") as f: tmp = manifest + ".tmp"
with open(tmp, "w") as f:
for r in records: for r in records:
f.write(json.dumps(r) + "\n") f.write(json.dumps(r) + "\n")
os.replace(tmp, manifest)
print(f"\nManifest: {manifest} ({len(records)} entries)") print(f"\nManifest: {manifest} ({len(records)} entries)")
# Quick checksum manifest for downstream debugging # Quick checksum manifest for downstream debugging
scripts/staffing/render_role_pool.py
+230
View File
@@ -0,0 +1,230 @@
#!/usr/bin/env python3
"""
render_role_pool.py — pre-render a role-aware face pool by hitting
serve_imagegen.py (localhost:3600/generate) with prompts pulled from
the bun server's /headshots/_scenes endpoint (single source of truth
for SCENES + SCENES_VERSION).
Layout:
data/headshots_role_pool/
{band}/
{gender}_{race}/
face_00.webp
face_01.webp
...
manifest.jsonl
Each entry in manifest.jsonl:
{"band": "warehouse", "gender": "man", "race": "caucasian",
"file": "warehouse/man_caucasian/face_03.webp",
"seed": 184729338, "scenes_version": "v1"}
Idempotent: a file at the target path is skipped. Re-run with --force
to regenerate. SCENES_VERSION is captured per render so the server's
pool route can refuse stale renders if the version drifts.
"""
from __future__ import annotations
import argparse
import base64
import json
import os
import sys
import time
import urllib.request
import urllib.error
DEFAULT_BANDS = ["warehouse", "production", "trades", "driver", "lead"]
DEFAULT_GENDERS = ["man", "woman"]
DEFAULT_RACES = ["caucasian", "east_asian", "south_asian", "middle_eastern", "black", "hispanic"]
def race_text(r: str) -> str:
return {
"caucasian": "",
"east_asian": "East Asian",
"south_asian": "South Asian",
"middle_eastern": "Middle Eastern",
"black": "Black",
"hispanic": "Hispanic",
}.get(r, "")
def fetch_scenes(mcp_url: str) -> tuple[str, dict]:
"""Pull canonical SCENES from the bun server. Single source of truth."""
req = urllib.request.Request(f"{mcp_url}/headshots/_scenes")
with urllib.request.urlopen(req, timeout=10) as resp:
data = json.loads(resp.read())
return data["version"], data["scenes"]
def render(comfy_url: str, prompt: str, seed: int, steps: int, timeout: int, dim: int) -> bytes | None:
payload = json.dumps({
"prompt": prompt,
"width": dim,
"height": dim,
"steps": steps,
"seed": seed,
}).encode()
req = urllib.request.Request(
f"{comfy_url}/generate",
data=payload,
headers={"Content-Type": "application/json"},
)
try:
with urllib.request.urlopen(req, timeout=timeout) as resp:
data = json.loads(resp.read())
except urllib.error.HTTPError as e:
print(f" HTTP {e.code} from comfy: {e.read()[:200]}", file=sys.stderr)
return None
except Exception as e:
print(f" comfy error: {type(e).__name__}: {e}", file=sys.stderr)
return None
img_b64 = data.get("image")
if not img_b64:
print(f" comfy response missing 'image' field: {list(data.keys())}", file=sys.stderr)
return None
return base64.b64decode(img_b64)
def main():
p = argparse.ArgumentParser()
p.add_argument("--out", default=os.path.join(os.path.dirname(__file__), "..", "..", "data", "headshots_role_pool"))
p.add_argument("--per-bucket", type=int, default=10, help="how many faces per (band × gender × race)")
p.add_argument("--mcp", default="http://localhost:3700")
p.add_argument("--comfy", default="http://localhost:3600")
p.add_argument("--steps", type=int, default=8)
p.add_argument("--bands", nargs="*", default=DEFAULT_BANDS)
p.add_argument("--genders", nargs="*", default=DEFAULT_GENDERS)
p.add_argument("--races", nargs="*", default=DEFAULT_RACES)
p.add_argument("--force", action="store_true", help="regenerate existing files")
p.add_argument("--age", type=int, default=32)
p.add_argument("--timeout", type=int, default=120, help="per-render timeout (1024² takes ~5s on A4000)")
p.add_argument("--dim", type=int, default=1024, help="square render dimension (v2 default 1024, v1 was 512)")
args = p.parse_args()
out_root = os.path.realpath(args.out)
os.makedirs(out_root, exist_ok=True)
print(f"Fetching canonical SCENES from {args.mcp}/headshots/_scenes…")
try:
version, scenes = fetch_scenes(args.mcp)
except Exception as e:
print(f"FATAL: could not fetch scenes ({e}). Is the mcp-server up?", file=sys.stderr)
sys.exit(1)
print(f" SCENES_VERSION={version}, {len(scenes)} bands available: {list(scenes.keys())}")
# v2+ files live at {out}/{version}/{band}/{g}_{r}/face_NN.webp.
# v1 lived at {out}/{band}/... — keep that layout intact for
# rollback; the server route reads both and prefers current.
out = out_root if version == "v1" else os.path.join(out_root, version)
os.makedirs(out, exist_ok=True)
print(f" writing to: {out}")
print(f" render dim: {args.dim}×{args.dim}")
# Reject any --bands not in the server's SCENES
unknown = [b for b in args.bands if b not in scenes]
if unknown:
print(f"FATAL: unknown bands {unknown}. Server has: {list(scenes.keys())}", file=sys.stderr)
sys.exit(1)
manifest_rows = []
todo = [
(band, g, r, n)
for band in args.bands
for g in args.genders
for r in args.races
for n in range(args.per_bucket)
]
print(f"\nPlanning: {len(todo)} renders ({len(args.bands)} bands × {len(args.genders)} genders × {len(args.races)} races × {args.per_bucket} faces).")
print(f"Estimated GPU time at 1.5s/render = {len(todo) * 1.5 / 60:.1f} min.\n")
t0 = time.time()
rendered = 0
skipped = 0
failed = 0
for i, (band, g, r, n) in enumerate(todo):
bucket_dir = os.path.join(out, band, f"{g}_{r}")
os.makedirs(bucket_dir, exist_ok=True)
fname = f"face_{n:02d}.webp"
full = os.path.join(bucket_dir, fname)
rel = os.path.relpath(full, out)
if os.path.exists(full) and os.path.getsize(full) > 1024 and not args.force:
skipped += 1
manifest_rows.append({
"band": band, "gender": g, "race": r, "file": rel,
"seed": None, "scenes_version": version, "cached": True,
})
continue
scene_def = scenes[band]
scene_clause = scene_def["scene"]
race_clause = race_text(r)
gender_clause = g # "man" / "woman"
# Match the bun server's prompt builder exactly. If you tweak
# one, tweak the other (or factor a /prompt-builder endpoint).
# The {role} slot is intentionally a band-typical title here
# — the pre-rendered face is shared across roles in the same
# band, so we use the band's archetypal role. Specific roles
# still hit the on-demand /headshots/generate/:key path with
# their actual title.
archetype_role = {
"warehouse": "warehouse worker",
"production": "production worker",
"trades": "skilled tradesperson",
"driver": "delivery driver",
"lead": "shift supervisor",
}.get(band, "warehouse worker")
prompt = (
f"professional headshot portrait of a {args.age}-year-old "
f"{race_clause} {gender_clause} {archetype_role}, {scene_clause}, "
f"neutral confident expression, sharp focus, photorealistic"
)
# Deterministic seed per slot — same (band, g, r, n) always
# gets the same face. Mixing scenes_version means a SCENES
# tweak shifts every face slightly; that's the right behavior
# (it's how cache invalidation propagates to the pool too).
seed_str = f"{band}|{g}|{r}|{n}|{version}"
seed_h = 5381
for ch in seed_str:
seed_h = ((seed_h << 5) + seed_h + ord(ch)) & 0x7fffffff
seed = seed_h
bytes_ = render(args.comfy, prompt, seed, args.steps, args.timeout, args.dim)
if bytes_ is None:
failed += 1
continue
with open(full, "wb") as f:
f.write(bytes_)
rendered += 1
manifest_rows.append({
"band": band, "gender": g, "race": r, "file": rel,
"seed": seed, "scenes_version": version, "cached": False,
})
if (i + 1) % 10 == 0 or (i + 1) == len(todo):
elapsed = time.time() - t0
done = i + 1
rate = done / elapsed if elapsed > 0 else 0
eta = (len(todo) - done) / rate if rate > 0 else 0
print(f" [{done}/{len(todo)}] rendered={rendered} skipped={skipped} failed={failed} "
f"rate={rate:.2f}/s eta={eta:.0f}s")
# Atomic manifest write
manifest_path = os.path.join(out, "manifest.jsonl")
tmp = manifest_path + ".tmp"
with open(tmp, "w") as f:
for row in manifest_rows:
f.write(json.dumps(row) + "\n")
os.replace(tmp, manifest_path)
print(f"\nDone. {rendered} new, {skipped} cached, {failed} failed in {time.time()-t0:.1f}s")
print(f"Manifest: {manifest_path} ({len(manifest_rows)} entries)")
print(f"\nNext: poke {args.mcp}/headshots/__reload to pick up the new pool.")
if __name__ == "__main__":
main()
scripts/staffing/tag_face_pool.py
+169
View File
@@ -0,0 +1,169 @@
#!/usr/bin/env python3
"""
tag_face_pool.py — run deepface gender + race classification over the
synthetic face pool produced by fetch_face_pool.py and rewrite
manifest.jsonl with `gender` (man / woman) and `race` (asian / black /
hispanic / indian / middle_eastern / white) tags.
Run with the venv that has deepface installed:
/home/profit/.local/share/deepface-venv/bin/python \
scripts/staffing/tag_face_pool.py
Idempotent: rows that already have BOTH gender and race tagged are
skipped. Pass --force to re-tag everything.
Mapping deepface buckets → /headshots/ ?e= values:
asian → split by manual region (deepface doesn't differentiate
East / South Asian; we lump as 'east_asian' since the
StyleGAN training set leans East Asian)
indian → south_asian
middle eastern → middle_eastern
black → black
hispanic → hispanic
white → caucasian
"""
from __future__ import annotations
import argparse
import json
import os
import sys
import time
DEEPFACE_RACE_TO_HINT = {
"asian": "east_asian",
"indian": "south_asian",
"middle eastern": "middle_eastern",
"black": "black",
"latino hispanic": "hispanic",
"hispanic": "hispanic",
"white": "caucasian",
}
def main():
p = argparse.ArgumentParser()
p.add_argument(
"--out",
default=os.path.join(os.path.dirname(__file__), "..", "..", "data", "headshots"),
)
p.add_argument("--force", action="store_true", help="re-tag rows that already have gender+race")
p.add_argument("--limit", type=int, default=0, help="cap how many faces to process this run (0 = all)")
p.add_argument("--min-age", type=int, default=22, help="exclude faces estimated below this age (kids/teens). Staffing context = legal-age workers only.")
args = p.parse_args()
out = os.path.realpath(args.out)
manifest_path = os.path.join(out, "manifest.jsonl")
if not os.path.exists(manifest_path):
print(f"manifest not found: {manifest_path}", file=sys.stderr)
sys.exit(1)
print(f"loading deepface (cold start ~10-15s for first model build)…")
from deepface import DeepFace # type: ignore
rows = []
with open(manifest_path) as f:
for line in f:
line = line.strip()
if not line:
continue
rows.append(json.loads(line))
print(f"manifest: {len(rows)} rows")
todo = [
r for r in rows
if args.force or r.get("gender") is None or r.get("race") is None or r.get("age") is None
]
if args.limit > 0:
todo = todo[: args.limit]
print(f"to tag: {len(todo)} faces")
if not todo:
print("nothing to do.")
return
counts_g = {}
counts_r = {}
failed = 0
t0 = time.time()
for i, r in enumerate(todo):
full = os.path.join(out, r["file"])
try:
ana = DeepFace.analyze(
img_path=full,
actions=["gender", "race", "age"],
enforce_detection=False,
silent=True,
)
if isinstance(ana, list):
ana = ana[0] if ana else {}
g_raw = (ana.get("dominant_gender") or "").lower().strip()
r["gender"] = (
"man" if g_raw.startswith("man") else
"woman" if g_raw.startswith("woman") else
None
)
r_raw = (ana.get("dominant_race") or "").lower().strip()
r["race"] = DEEPFACE_RACE_TO_HINT.get(r_raw, None)
if r["race"] is None and r_raw:
r["race_raw"] = r_raw
# Age estimation — exclude minors / teens. Staffing context
# uses adult workers only. Threshold is 22 by default
# (legal + a buffer because age estimation is noisy).
try:
age = int(round(float(ana.get("age") or 0)))
except Exception:
age = 0
r["age"] = age
if age and age < args.min_age:
r["excluded"] = "minor"
else:
r.pop("excluded", None)
counts_g[r["gender"] or "unknown"] = counts_g.get(r["gender"] or "unknown", 0) + 1
counts_r[r["race"] or r_raw or "unknown"] = counts_r.get(r["race"] or r_raw or "unknown", 0) + 1
except Exception as e:
r["tag_error"] = f"{type(e).__name__}: {e}"
failed += 1
if (i + 1) % 25 == 0 or (i + 1) == len(todo):
elapsed = time.time() - t0
rate = (i + 1) / elapsed if elapsed > 0 else 0
eta = (len(todo) - i - 1) / rate if rate > 0 else 0
print(f" [{i+1}/{len(todo)}] rate={rate:.1f}/s eta={eta:.0f}s failed={failed}")
print(f" gender: {counts_g}")
print(f" race : {counts_r}")
# Write updated manifest atomically
tmp = manifest_path + ".tmp"
with open(tmp, "w") as f:
for r in rows:
f.write(json.dumps(r) + "\n")
os.replace(tmp, manifest_path)
final_g = {}
final_r = {}
excluded = 0
age_hist = {"<18": 0, "18-22": 0, "22-30": 0, "30-40": 0, "40-50": 0, "50-60": 0, "60+": 0, "unknown": 0}
for r in rows:
if r.get("excluded"):
excluded += 1
continue
final_g[r.get("gender") or "untagged"] = final_g.get(r.get("gender") or "untagged", 0) + 1
final_r[r.get("race") or "untagged"] = final_r.get(r.get("race") or "untagged", 0) + 1
a = r.get("age") or 0
if a == 0: age_hist["unknown"] += 1
elif a < 18: age_hist["<18"] += 1
elif a < 22: age_hist["18-22"] += 1
elif a < 30: age_hist["22-30"] += 1
elif a < 40: age_hist["30-40"] += 1
elif a < 50: age_hist["40-50"] += 1
elif a < 60: age_hist["50-60"] += 1
else: age_hist["60+"] += 1
print(f"\nDone. {len(rows)} rows, {excluded} excluded as <{args.min_age}, {failed} tag errors, {time.time()-t0:.1f}s")
print(f" final gender: {final_g}")
print(f" final race : {final_r}")
print(f" age dist : {age_hist}")
print(f"\nNext: poke /headshots/__reload to refresh the in-memory pool.")
if __name__ == "__main__":
main()
tests/real-world/scrum_master_pipeline.ts
+13 -2
View File
@@ -1143,9 +1143,15 @@ Format each as a code-fenced block with the byte offset within the shard:
EXACT LINE OF SOURCE — DO NOT PARAPHRASE, DO NOT TRUNCATE EXACT LINE OF SOURCE — DO NOT PARAPHRASE, DO NOT TRUNCATE
\`\`\` \`\`\`
Pick the most reviewer-relevant lines: route definitions (e.g. \`@app.route(...)\`), function signatures, security-sensitive calls (auth/SQL/exec/template/secrets), hardcoded credentials/defaults, exception handlers, sensitive imports. The reviewer will REFUSE to act on any claim not backed by a verbatim anchor — so anchors are how you prove findings are real.`; Pick the most reviewer-relevant lines: route definitions (e.g. \`@app.route(...)\`), function signatures, security-sensitive calls (auth/SQL/exec/template/secrets), hardcoded credentials/defaults, exception handlers, sensitive imports. The reviewer will REFUSE to act on any claim not backed by a verbatim anchor — so anchors are how you prove findings are real.`;
// 2026-04-28: gpt-oss:120b → gemini-3-flash-preview via Ollama
// Pro. Tree-split MAP fires once per shard (potentially 5-20×
// per file), so latency dominates total scrum time. Gemini 3
// flash returns shard digests substantially faster than the old
// 120B free model while staying strong enough for byte-anchored
// extraction.
const r = await chat({ const r = await chat({
provider: "ollama_cloud", provider: "ollama_cloud",
model: "gpt-oss:120b", model: "gemini-3-flash-preview",
prompt, prompt,
max_tokens: 900, max_tokens: 900,
}); });
@@ -1195,9 +1201,14 @@ COPY EVERY anchor block from the piece notes IN ORDER, character-perfect. DO NOT
Output the anchor blocks under their original \`\`\`@offset...\`\`\` fences, each on its own with a blank line between. The reviewer rejects findings that don't quote a string from this anchors block, so completeness here directly determines review quality.`; Output the anchor blocks under their original \`\`\`@offset...\`\`\` fences, each on its own with a blank line between. The reviewer rejects findings that don't quote a string from this anchors block, so completeness here directly determines review quality.`;
// 2026-04-28: gpt-oss:120b → gemini-3-flash-preview via Ollama
// Pro. The reducer runs once per file (vs once per shard for MAP)
// but on a much larger context (all shard digests stacked), so
// throughput per token still matters. Same model as MAP for
// consistency in tree-split outputs.
const reduced = await chat({ const reduced = await chat({
provider: "ollama_cloud", provider: "ollama_cloud",
model: "gpt-oss:120b", model: "gemini-3-flash-preview",
prompt: reducePrompt, prompt: reducePrompt,
max_tokens: 2400, max_tokens: 2400,
}); });