Compare commits

..
17 Commits
Author SHA1 Message Date
root 5225211e45 demo: real synthetic headshots — fetch pool + serve route + UI wire
Three layers shipped:

1. SCRIPT — scripts/staffing/fetch_face_pool.py
   Pulls N synthetic StyleGAN faces from thispersondoesnotexist.com
   into data/headshots/face_NNNN.jpg, writes manifest.jsonl. Idempotent:
   re-running skips existing files. Optional gender tagging via deepface
   (currently unavailable on this box; the script handles ImportError
   gracefully and tags everything as untagged). Fetched 198 faces with
   concurrency=3 in ~67s.

2. SERVER — /headshots/:key route in mcp-server/index.ts
   Loads manifest at first hit, caches in globalThis._faces. Hashes the
   key with djb2-style mixing → pool index → returns the JPG. Same
   key always gets the same face (deterministic). Accepts
   ?g=man|woman&e=caucasian|black|hispanic|south_asian|east_asian|middle_eastern
   to bias pool selection — the gender/ethnicity buckets fall back to
   the full pool when no tagged matches exist. Cache-Control:
   86400 immutable so faces ride the browser cache after first hit.
   /headshots/__reload re-reads the manifest without restart.

3. UI — search.html + console.html worker cards
   Re-added overlay <img> on top of the monogram .av circle. img.src
   = /headshots/<encoded-key>?g=<hint>&e=<hint>. img.onerror removes
   the failed image so the monogram stays visible if the face pool
   isn't fetched / CDN is blocked. .av now has overflow:hidden +
   position:relative to clip the img to a perfect circle.

Forced-confident name resolution (J: "we're CREATING the profile,
created as though you truly have the information Xavier is more
likely Hispanic and he's a male"):

   genderFor(name)        — looks up MALE_NAMES + FEMALE_NAMES,
                            falls back to a deterministic hash split
                            so unknown names spread ~50/50. Sets now
                            include cross-cultural names: Alejandro/
                            Andres/Mateo/Santiago/Joaquin/Cesar/Hugo/
                            Felipe/Gerardo/Salvador/Ramon (Hispanic),
                            Raj/Anil/Vikram/Krishna/Pradeep (South
                            Asian), Wei/Yi/Hiroshi/Akira/Hyun (East
                            Asian), Demetrius/Kareem/DaQuan/Khalil
                            (Black), Omar/Khalid/Hassan/Ahmed/Bilal
                            (Middle Eastern). FEMALE_NAMES extended
                            in parallel.

   guessEthnicityFromFirstName(name)
                          — confident default of 'caucasian' for any
                            name not in the cultural buckets so every
                            worker resolves to a category the face
                            pool can be biased toward. Order: ME → Black
                            → Hispanic → South Asian → East Asian →
                            Caucasian (matters where names overlap,
                            e.g. Aisha appears in ME + Black, biases
                            toward ME for visual fit).

   Both helpers also ported into console.html so the triage backfills
   and try-it-yourself rendering get the same hint stack.

Privacy note in the script + route comments: the synthetic data uses
the worker's name as the seed; production should hash worker_id (not
name) to avoid leaking PII to a third-party CDN. The fetch URL itself
is referenced once per pool build, not per-worker.

.gitignore — added data/headshots/face_*.jpg (~100MB for 198 faces;
the manifest + script are tracked). Re-running the script on a fresh
checkout rebuilds the pool from scratch.

Verified end-to-end via playwright on devop.live/lakehouse:
   forklift query → 10 worker cards
   10/10 with face images (real synthetic headshots, not monograms)
   0/10 broken
   Alejandro G. Nelson  → ?g=man&e=hispanic
   Patricia K. Garcia    → ?g=woman&e=caucasian
   Each name → unique face, deterministic across loads.
   Console triage backfills get the same treatment.
2026-04-28 00:04:03 -05:00
root 4f0b6fb9b3 demo: console — sober worker cards (mirror dashboard styling)
J: "can you update Staffer's Console too the same look." Console
rendered worker rows in three places (Chapter 4 permit-contract
candidates, Chapter 8 triage backfills, Chapter 9 try-it-yourself
results) with the original 28px square avatar + flat backgrounds —
inconsistent with the new dashboard design.

Three changes:

1. CSS — .worker now has a 3px left-edge border that color-codes the
   role family, and .av is a 32px circle with a muted dark background
   + 1px ring + monogram initials. Five role-band colors mirror
   search.html: warehouse blue / production amber / trades purple /
   driver green / lead orange. Plus a .role-pill style matching the
   dashboard's small uppercase chip.

2. Helpers — added ROLE_BANDS regex table + roleBand() classifier and
   a new workerRow(name, role, detail, opts) builder. Same regex
   patterns as search.html so a "Forklift Operator" classifies
   identically on every page. opts.endorsed adds the green endorsed
   chip; opts.score appends a rank badge.

3. Replaced the three inline avatar+row constructors with workerRow()
   calls. Net: console.html lost ~20 lines of duplicated DOM building
   while gaining role bands + pills.

Verified end-to-end via playwright on devop.live/lakehouse/console:
  Chapter 8 triage scenario "Marcus running late site 4422":
    5 backfill rows render with [warehouse] band + WAREHOUSE pill +
    monogram avatars (SBC, ETW, SHC, WMG, MEB).
  Same sober look as the dashboard worker cards. No emojis, no
  cartoons, color-coded role family on the left edge.
2026-04-27 23:47:12 -05:00
root 8e781ac325 demo: worker cards — sober monogram avatars + role bands (no cartoons)
J: "It's two cartoonish right now the website looks like it was made
by first grade teacher." Pulled the DiceBear personas-style headshots
and the emoji role badges. They were generative-illustration playful;
this is supposed to read like a staffing tool, not a kindergarten
attendance sheet.

Replacement design — restraint, signal, no glyphs:

  Avatar:   40px circle, monogram initials, muted dark background
            (#161b22), 1px ring (#21262d), white-ish text. No image,
            no emoji. Looks like a pre-photo placeholder slot in a
            real ATS.

  Role band: the role gets classified into one of five families:
            WAREHOUSE / PRODUCTION / SKILLED TRADE / DRIVER / LEAD
            (regex-based; falls back to the first word of the role
            for unknown families). Each family has a single muted
            color: blue / amber / purple / green / orange. The
            color appears as:
              - a 3px left border on the .iworker card
              - a 2px left border + matching text color on a small
                uppercase pill in the detail line

That's it. No images, no emojis, no per-role illustrations. The
staffer sees role-family at a glance via the band color, name and
initials prominently, full role + city + zip in the detail string
behind the pill. Five colors total instead of an eight-color rainbow.

CSS:
  .iworker[data-role-band="warehouse"] etc. → 3px left border
  .role-pill[data-rb="warehouse"] etc.      → matching pill border

JS:
  ROLE_BANDS = 6 regex → band+label entries (warehouse, production,
                          trades, driver, lead, quality)
  roleBand(role)       = first matching entry, fallback to first
                          word of role uppercased

Verified end-to-end via playwright on devop.live/lakehouse:
  forklift query → 10 cards
  every card → monogram avatar + WAREHOUSE pill (blue band)
  no images, no emojis, no rainbow

Restart sequence after these edits:
  pkill -9 -f "/home/profit/lakehouse/mcp-server/index.ts"
  ( setsid bun run /home/profit/lakehouse/mcp-server/index.ts \
      > /tmp/mcp-server.log 2>&1 < /dev/null & disown )
2026-04-27 23:43:36 -05:00
root ee0450b7c3 demo: spec — refresh repo layout + model fleet + per-staffer + paths 8-9
J: "how about devop.live/lakehouse/spec." Spec was anchored on
2026-04-21 state (v2 footer): mistral mentioned in the model matrix,
13 crates not 15, missing validator/truth/auditor crates, no mention
of OpenCode 40-model fleet, no pathway memory, no per-staffer
hot-swap, no Construction Activity Signal Engine, ADR count was 20.
Footer claimed Phases 19-25.

Edits, in order:

  Ch1 Repository layout
    + crates/truth/ (ADR-021 rule store)
    + crates/validator/ (Phase 43 — schema/completeness/policy gates)
    + auditor/ (cross-lineage Kimi↔Haiku/Opus auto-promote)
    + scripts/distillation/ (frozen substrate v1.0.0 at e7636f2)
    Updated aibridge to mention ProviderAdapter dispatch
    Updated gateway to mention OpenAI-compat /v1/* drop-in middleware
    Updated mcp-server route list to include /staffers + profiler/contractor pages
    Updated config/ to mention modes.toml + providers.toml + routing.toml
    Updated docs/ ADR count from 20 → 21
    Updated data/ to mention _pathway_memory + _auditor/kimi_verdicts

  Ch3 Measurement & indexing
    REPLACED stale "Model matrix (Phase 20)" T1-T5 table that
    mentioned mistral with the current 5-provider fleet:
      ollama / ollama_cloud / openrouter / opencode (40 models, one
      sk-* key reaches Claude Opus 4.7, GPT-5.5-pro, Gemini 3.1-pro,
      Kimi K2.6, GLM, DeepSeek, Qwen, MiniMax, free) / kimi
    ADDED 9-rung cloud-first ladder pseudocode
    ADDED N=3 consensus + cross-architecture tie-breaker math
    ADDED auditor cross-lineage Kimi K2.6 ↔ Haiku 4.5 + Opus auto-promote
    ADDED distillation v1.0.0 freeze paragraph (145 tests, 22/22, 16/16)
    Updated Continuation primitive to mention Phase 44 Rust port

  Ch5 What a CRM can't do
    Extended the table with 6 new capabilities:
      - Per-staffer relevance gradient
      - Triage in one shot (late-worker → backfills + draft SMS)
      - Permit → fill plan derivation
      - Public-issuer attribution across contractor graph
      - Cross-lineage AI audit on every PR
      - Pathway memory (system-level hot-swap, ADR-021)

  Ch6 How it gets better over time
    Lede updated from 7 paths → 10 paths
    NEW Path 7 — Pathway memory (ADR-021)
    NEW Path 8 — Per-staffer hot-swap index
    NEW Path 9 — Construction Activity Signal Engine
    Original Path 7 (observer ingest) renumbered to Path 10

  Ch9 Per-staffer context
    Lede now anchors Path 8 from Ch6
    NEW lead section: Per-staffer hot-swap index — Maria/Devon/Aisha,
    same query reshapes per coordinator (167 IL / 89 IN / 16 WI),
    MARIA'S MEMORY pill, /staffers endpoint, metro-agnostic by
    construction. The original Phase 17 profile / Phase 23 competence
    sections retained beneath as the deeper architecture detail.

  Ch10 A day in the life
    Updated 14:00 emergency event to use the late-worker triage
    handler — coordinator types "Dave running late site 4422", gets
    profile + draft SMS + 5 backfills + Copy SMS button in 250ms.
    The old Click No-show button → /log_failure flow remains valid
    (penalty still records); the user-facing surface is the new
    triage card.

  Ch11 Known limits
    REPLACED the Mem0/Letta/Phase-26 era list with current honest
    limits: BAI persistence + backtesting, NYC DOB adapter, 12
    awaiting public-data sources for contractor profile, rate/margin
    awareness, Mem0-style UPDATE/DELETE, Letta hot cache (now 5K
    not 1.9K), confidence calibration, SEC fuzzy precision, tighter
    pathway+scrum integration.

  Footer
    v2 2026-04-21 → v3 2026-04-27
    Phases 19-25 → 19-45
    Lists today's phases: distillation v1.0.0 substrate, gateway as
    OpenAI-compat drop-in, mode runner, validator + iterate, ADR-021
    pathway memory, per-staffer hot-swap, Construction Activity Signal
    Engine.

  Nav
    + Profiler link
    Date pill v1 · 2026-04-20 → v3 · 2026-04-27

Verified end-to-end on devop.live/lakehouse/spec — 11 chapter h2s
render in order, 67KB page (was 50KB-ish), all internal links resolve.
2026-04-27 23:22:07 -05:00
root 0f9b4aa2fe demo: proof — full architecture-page rewrite for current state
J: "needs a rewrite." Old version was anchored on a dual-agent
mistral+qwen2.5 loop that hasn't been the model story for weeks,
called the system 13 crates (it's 15), referenced "Local 7B models"
in the honest-limits section, and had no mention of:
  - the 40-model OpenCode fleet via one sk-* key
  - the 9-rung cloud-first ladder
  - N=3 consensus + cross-architecture tie-breaker
  - auditor cross-lineage (Kimi K2.6 ↔ Haiku 4.5, Opus auto-promote)
  - distillation v1.0.0 frozen substrate (e7636f2)
  - pathway memory (88 traces, 11/11 replays, ADR-021)
  - per-staffer hot-swap index
  - Construction Activity Signal Engine + BAI + ticker network
  - the gateway as OpenAI-compat drop-in middleware

Rewrote into 10 chapters:

  1.  Receipts — live tests + new live tile showing the Signal Engine
      view for THIS load (issuer count, attributed build value,
      contractor count, attribution edges)
  2.  Architecture — corrected to 15 crates with current responsibilities;
      ASCII diagram showing OpenAI consumers + MCP + Browser all hitting
      gateway /v1/*; provider fleet table with all 5 (ollama, ollama_cloud,
      openrouter, opencode 40-model, kimi); validator + truth + auditor
      crates added
  3.  Model fleet — REPLACED the dual-agent mistral story. Now: the
      9-rung ladder (kimi-k2:1t through openrouter:free → ollama local),
      N=3 consensus + tie-breaker math, auditor Kimi↔Haiku alternation
      with Opus auto-promote on big diffs, distillation v1.0.0 freeze
      tag e7636f2 (145 tests · 22/22 · 16/16 · bit-identical)
  4.  Two memory layers — kept playbook content (Phase 19 boost math
      still load-bearing), added pathway memory (ADR-021) section with
      live counters in the page (88 / 11-11 / 100% reuse rate)
  5.  Per-staffer hot-swap — NEW. Pseudocode showing how staffer_id
      scopes state filter + playbook geo + UI relabel to MARIA'S MEMORY
  6.  Construction Activity Signal Engine — NEW. Three attribution
      flavors (direct, parent, associated), BAI math, cross-metro
      replication framing (NYC DOB next, then LA / Houston / Boston)
  7.  Architectural choices — added ADR-021 row + distillation freeze row
  8.  Measured at scale — kept (uses /proof.json scale data)
  9.  Verify or dispute — REFRESHED with current endpoints. Removed the
      stale "bun run tests/multi-agent/scenario.ts" recipe; added curl
      examples for /v1/health, pathway/stats, per-staffer scoping (3-loop
      bash script), late-worker triage, profiler_index, ticker_quotes,
      auditor verdicts, distillation acceptance gate
  10. What we are NOT claiming — REFRESHED. Removed "Local 7B models"
      caveat; added: 12 awaiting public-data sources are placeholders,
      SEC name-fuzzy has rare false positives, BAI is a thesis not a
      backtest yet, single-metro today

Live data probes added:
  loadPathwayLive   — fills pwm-traces / pwm-replays / pwm-rate spans
  loadSignalLive    — renders the LIVE Signal Engine tile under Ch1

Nav also gained a Profiler link to match search.html and console.html.

Verified end-to-end on devop.live/lakehouse/proof:
  10 chapters render, 5/5 live tests pass, pathway shows 88 traces +
  100% reuse rate, live signal tile shows 11 issuers + $347M attributed
  + 200 contractors + 14 attribution edges. Architecture diagram +
  crate table accurate as of HEAD.
2026-04-27 23:13:46 -05:00
root e4eb0fa168 demo: console — three new chapters reflecting recent shipments
J: "it's outdated." Console walkthrough was stuck on the original 6
chapters (legacy-bridge / permits / catalog / ranking demo / playbook
memory / try-it-yourself). Three weeks of new work weren't visible.

Three new chapters added between the existing playbook-memory chapter
and the input box; all pull live data from the running system:

  Chapter 6 — Three coordinators, three views of the same corpus
    Renders Maria/Devon/Aisha cards from /staffers with their
    territories. Frames the per-staffer hot-swap as the relevance
    gradient that compounds independently per coordinator. Same query
    "forklift operators" returns 89 IN / 16 WI / 167 IL workers
    depending on who's acting.

  Chapter 7 — The hidden signal — public issuers in your contractor graph
    Pulls /intelligence/profiler_index, builds the basket, shows
    issuer count + attributed build value + contractor count as the
    three top metrics. Lists top 8 issuers with attribution counts
    and direct-link to the profiler. This is the BAI / Signal Engine
    pitch in walkthrough form: every contractor name is also a forward
    indicator on a public equity. Cross-metro replication explicit
    in the closing paragraph.

  Chapter 8 — When something breaks — triage in one shot
    Live triage demo against /intelligence/chat with body
    {message:"Marcus running late site 4422"}. Renders the worker
    card + draft SMS + 5 backfills + duration_ms. The 250ms-vs-20min
    moment, made concrete with real Quincy IL workers.

Chapter 9 (was 6) — Try it yourself
  Updated input examples to demonstrate each new route:
    "8 production workers near 60607" → headcount + zip parser
    "Marcus running late site 4422"  → triage handler
    "Marcus"                          → bare-name lookup
    "what came in last night"         → temporal route
    "reliable forklift operators with OSHA certs" → hybrid SQL+vector
  Each is a click-to-run link beneath the input.

Two new accent classes: .accent-g (green for issuer-count) and
.accent-r (red for triage event).

Verified end-to-end on devop.live/lakehouse/console: 9 chapters
render, ch6 shows 3 staffer personas, ch7 shows 11 issuers / $347M /
200 contractors, ch8 shows Marcus V. Campbell + draft SMS + 5
backfills.
2026-04-27 23:04:37 -05:00
root 93081bed5c ops: persist runtime fixes — iterate.rs unused state, catalog cleanup
Two load-bearing runtime changes that were never committed:

1. crates/gateway/src/v1/iterate.rs — `state` → `_state` on the unused
   route-state parameter. Cleared the one cargo workspace warning.
   Fix was made earlier this session but the working-tree change
   never made it into a commit.

2. data/_catalog/manifests/564b00ae-cbf3-4efd-aa55-84cdb6d2b0b7.json —
   DELETED. This was the dead manifest for `client_workerskjkk`, a
   typo dataset whose parquet was deleted but whose catalog entry
   stayed registered. Every SQL query failed schema inference on the
   missing file before reaching its target table — that's the bug
   that made /system/summary report 0 workers and the demo show zero
   bench. Deleting the manifest keeps the fix on disk; committing
   the deletion keeps it in git so a fresh checkout doesn't regress.

3. data/_catalog/manifests/32ee74a0-59b4-4e5b-8edb-70c9347a4bf3.json
   — runtime catalog metadata update from the successful_playbooks_live
   write path. Ride-along change.

Reports under reports/distillation/phase[68]-*.md are auto-regenerated
by the audit cycle each run; skipping those.
2026-04-27 23:00:16 -05:00
root 885a1acf19 demo: System Activity panel — capability index reflects every recent shipment
Old panel showed playbook ops + search counts and went empty in a
fresh demo (no operations yet). J: "update System Activity to coincide
with all of our recent updates."

Rebuilt as a live capability index — each tile is a thing the
substrate has learned to do, with the metric proving it's running.
Pulled in parallel from /staffers, /system/summary,
/api/vectors/playbook_memory/stats, /api/vectors/pathway/stats,
/intelligence/profiler_index, /intelligence/activity. Each probe
catches its own error so a single missing endpoint doesn't collapse
the panel.

Nine capability cards (verified end-to-end on devop.live/lakehouse):

  1. Per-staffer hot-swap index           3 personas (Maria/Devon/Aisha)
  2. Construction Activity Signal Engine  11 issuers · $347M attributed
                                          build value · network 11/14
  3. Late-worker / no-show triage         one-shot — name+late → backfills+SMS
  4. Permit → staffing bridge             24/day, every Chicago permit ≥$250K
  5. Hybrid SQL + vector search           500K workers · 5,474 playbook entries
  6. Schema-agnostic ingestion            36 datasets · 2.98M rows
  7. Contractor profile + project index   6 wired · 12 queued sources
  8. Pathway memory                       88 traces · 11/11 replays · 100%
  9. Ticker association network           11 tickers · 3 direct + 11 associated

Each card carries:
  - capability title + ship date pill ("baseline" or "shipped 2026-04-27")
  - big metric (live, not pre-baked)
  - sub-context line in coordinator language
  - "why a staffer cares" explanation
  - optional "Open →" deep link to the surface (Profiler, Contractor)

Header + intro paragraph reframed: "what the substrate has learned to
do" instead of "what the substrate has learned." Operational learning
(fills, playbooks, hot-swaps) compounds INSIDE each capability; the
panel surfaces the set of capabilities the corpus knows how to express.

Closing operational-stats row at the bottom shows fills/searches/
recent playbooks when /intelligence/activity has any.
2026-04-27 22:54:52 -05:00
root 97888e3775 demo: profiler — Construction Activity Signal Engine narrative + BAI
J's prompt: shoot for the stars, frame the data corpus's value as a
predictive signal, not just a contractor directory. The thesis is
that every name in this corpus is also a forward indicator on public
equities — permits filed today predict construction starts in ~45
days, staffing in ~30, revenue recognition months later. The
associated-ticker network surfaces this signal before any 10-Q does.

Two new layers above the basket:

1. HERO THESIS PANEL — "Chicago Construction Activity Signal Engine"
   header + 3-line value statement, then 4 live metrics:

   - BAI (Building Activity Index) — attribution-weighted average of
     day-change % across surfaced issuers. Weight = attribution count
     so issuers we have more depth on count more. Today: +0.76%
     (9 issuers · top contributors FCBC +2.4%, ACRE +1.7%, JPM +1.5%).
     Color-coded green/red.

   - Indexed build value — total $ of permits attributable to ANY
     public issuer in this view. Today: $344M.

   - Network depth — issuers / attribution edges. Today: 9 / 15.
     This is the "we see what nobody else sees" metric: how many
     contractors are bridges from a private builder back to a public
     equity holder.

   - Market replication roadmap — chips showing "Chicago — live ·
     NYC DOB — adapter ready · LA County · Houston BCD · Boston ISD
     · DC DCRA". Frames the corpus as metro-agnostic from day one.

2. PER-TICKER ACTIVITY MAP — when a basket card is clicked, a leaflet
   map appears below the basket plotting that ticker's geocoded permit
   activity. Pulls /intelligence/contractor_profile for up to 6
   attributed contractors, merges their geocoded permits, plots on a
   dark Chicago tile layer. Color-banded by permit cost (green <$100K,
   amber $100K-$1M, red ≥$1M). Click TGT → 23 Target permits across
   Chicago; click JPM → JPMorgan-adjacent contractor activity. Cached
   per ticker so toggling is instant.

Verified end-to-end on devop.live/lakehouse/profiler:
  Default load: hero panel renders with all 4 metrics, basket strip
                with 9 issuers + live prices in 669ms.
  Click TGT  : signal map activates, "23 geocoded permits across
                1 contractor", table filters to 2 rows.
  Tooltip on basket cards: full reason path including matched name +
                contributors attributed to that ticker.

Architecture-side: zero new server code — all metrics computed
client-side from the existing profiler_index + ticker_quotes payloads.
The corpus already had the value; the page just needed to articulate it.
2026-04-27 22:23:46 -05:00
root 9b8befaa94 demo: profiler — scrolling ticker basket with live prices + click-to-filter
J asked: "kind of like a scrolling ticker that has all of the companies
and their stock prices and where they fit in the map." Implemented as
a horizontal-scroll strip at the top of /profiler:

  9 public issuers in this view · quotes via Stooq · 669ms
  ┌────┬────┬────┬────┬────┬────┬────┐
  │TGT │JPM │BALY│ACRE│FCBC│NREF│LSBK│ ← live price + day-change per
  │129 │311 │... │... │... │... │... │   ticker, color-banded by
  │+.17│+1.5│... │... │... │... │... │   attribution kind
  └────┴────┴────┴────┴────┴────┴────┘

Each card carries:
  - ticker + live price + day-change % (red/green)
  - attribution count + kind (exact / direct / parent / associated)
  - left bar color = strongest attribution kind (green for direct
    issuer, amber for parent, blue for co-permit associated, gradient
    when both direct and associated apply)
  - tooltip on hover lists the contractors attributed to this ticker
  - click toggles a filter on the table below — clicking TGT cuts the
    200-row list down to just TARGET CORPORATION + TORNOW, KYLE F
    (Target's primary co-permit contractor)

Server-side:
- entity.ts exports fetchStooqQuote (was internal)
- new POST /intelligence/ticker_quotes — accepts {tickers: [...]},
  fans out to Stooq.us in parallel, returns
  {ticker, price, price_date, open, high, low, day_change_pct,
   stooq_url} per symbol or null for non-US listings (HOC.DE, SKA-B.ST,
   LLC.AX). Capped at 50 symbols per call.

Front-end:
- mcp-server/profiler.html — new .basket-wrap section above the
  controls. buildBasket() runs after profiler_index loads:
    1. Aggregates unique tickers from .tickers.direct + .associated
       across all surfaced contractors
    2. Renders shells immediately (ticker symbol + "—" placeholder)
    3. Batch-fetches quotes via /intelligence/ticker_quotes
    4. Updates each card with price + day-change in place
  Click on a card sets a tickerFilter; render() skips rows whose
  attributions don't include that ticker. "clear filter" button on
  the basket strip resets it.

Verified end-to-end on devop.live/lakehouse/profiler:
  Default load → 9 issuers, live prices populated in 669ms
  TGT click   → table filters to TARGET CORPORATION + TORNOW, KYLE F
                (the contractor who runs 3 of Target's recent permits
                gets the TGT correlation indicator)
  JPM card    → $311.63, +1.55% — JPMorgan-adjacent contractors
  Tooltip     → list of contractors attributed to the ticker
2026-04-27 22:19:26 -05:00
root 2965b68a9d demo: profiler index — ticker associations (direct, parent, co-permit)
J's framing: "if a contractor works for Target, future Target contracts
mean money flows back to the contractor — the ticker is an associated
indicator." Now the profiler index attaches three flavors of ticker per
contractor and renders them as colored pills:

  green DIRECT    contractor IS the public issuer (Target Corp → TGT)
  amber PARENT    contractor is a subsidiary of a public parent
                    (Turner Construction → HOC.DE via Hochtief AG)
  blue  ASSOCIATED contractor co-appears on permits with a public
                    entity (TORNOW, KYLE F → TGT, 3 shared permits with
                    TARGET CORPORATION)

The associated flavor is the correlation signal J described — it pulls
the ticker for whoever the contractor has been working *with*, not
just what they are themselves. Most contractors are private; the
associated link is how the moat shows up.

Server-side:
- entity.ts new export `lookupTickerLite(name)` — cheap in-memory
  resolver that does only the SEC tickers index lookup + curated
  KNOWN_PARENT_MAP check, no per-call SEC profile or Stooq fetch.
  ~10ms per name after the index is loaded once.
- /intelligence/profiler_index now runs a third Socrata pull
  (5K permit pairs in window) to build a co-occurrence map. For each
  contractor in the result, attaches:
    .tickers.direct[]      — name matches a public issuer
    .tickers.associated[]  — top 5 co-permit partners that resolve
                              to a ticker, with partner_name +
                              co_permits count + partner_via reason

Front-end:
- mcp-server/profiler.html — new .ticker-pill styles (3 colors per
  attribution kind), pills render under the contractor name in the
  table. Hover title gives the full reason path.

Verified end-to-end on the public URL:
  search="tornow" → blue TGT pill, hint "Associated via co-permits
                    with TARGET CORPORATION (3 shared permits) —
                    TARGET CORP"
  search="target" → green TGT × 2 (TARGET CORPORATION +
                    CORPORATION TARGET name variants both resolve
                    direct to the same issuer)
  default top 200 → 15 ticker pills surface across the page including
                    JPM (via JPMORGAN CHASE BANK co-permits) and
                    parent-link tickers for the construction majors.
2026-04-27 22:08:24 -05:00
root 08c8debfff demo: profiler index — directory of every Chicago contractor
J asked for "a profiler index that shows a history of everyone." This
is a /profiler directory page (also reachable via /contractors) that
ranks every contractor who's filed a Chicago permit, by total permit
value. Rows are clickable into the full /contractor profile.

Defaults: since 2025-06-01, min permit cost $250K, top 200 contractors
by total_cost. Server pulls two Socrata GROUP BY queries (one keyed on
contact_1_name, one on contact_2_name), merges them so contractors
listed in either applicant or contractor slot appear once with combined
counts/cost. ~300ms cold.

UI: live search box, since-date selector, min-cost selector, sortable
columns (name / permits / total_cost / last_filed). Live numbers as of
this write: 200 contractors, 1,702 permits, $14.22B aggregate. Filter
"Target" returns TARGET CORPORATION + CORPORATION TARGET (name variants
from Socrata).

Also fixes J's other complaint — "no new contracts, Target is gone":

  /intelligence/permit_contracts was hard-capped at $limit=6 + only
  the most recent 6 over $250K, so any day with 6 fresh permits would
  push older contractors (Target) off the panel entirely. Now defaults
  to 24 (caller can pass body.limit up to 100), so 2-3 days of permits
  stay on the panel. Added body.contractor — passes a name into the
  WHERE so the staffer can pin a specific contractor to the panel
  ("Target Corporation" → 3 of their permits over $250K).

Server-side:
- new POST /intelligence/profiler_index — paginated contractor index
  (since, min_cost, search, limit) with merged contact_1+contact_2
  aggregations
- /intelligence/permit_contracts — body.limit + body.contractor
- /profiler and /contractors routes serve profiler.html

Front-end:
- new mcp-server/profiler.html — sortable table, live filter, deep
  links to /contractor?name=... (prefix-aware via P, so /lakehouse
  works on devop.live)
- search.html + console.html nav: added "Profiler" link

Verified end-to-end via playwright on the public URL.
2026-04-27 22:00:52 -05:00
root b02cf5b9e1 demo: contractor links — respect the /lakehouse path prefix
J reported https://devop.live/contractor?name=3115%20W%20POLK%20ST.%20LLC
returned 404. Cause: the anchor href was a bare /contractor, which on
devop.live routes to the LLM Team UI (port 5000) at the main site root,
not the lakehouse mcp-server (which lives under /lakehouse/*).

Every page that renders a contractor link now uses the same prefix
detector the dashboard already had:

  var P = location.pathname.indexOf('/lakehouse') >= 0 ? '/lakehouse' : '';

Files updated:
- search.html: entity-brief anchor + preview anchor → P+/contractor
- console.html: permit-card contractor list → P+/contractor
- contractor.html: history.replaceState + back-link + the
  /intelligence/contractor_profile fetch all use P prefix. The page
  is reachable at /lakehouse/contractor on the public URL and bare
  /contractor on localhost; both work without further config.

Verified:
  https://devop.live/lakehouse/contractor?name=3115%20W%20POLK%20ST.%20LLC
    → 200, 29.9 KB, full profile renders. Contractor has 1 permit on
    file (a small LLC), 1 geocoded so the heat map plots one marker.
2026-04-27 21:44:59 -05:00
root 1ac8045924 demo: contractor profile — heat map, project index, 12 awaiting sources
The contractor.html click-target J asked for: a separate page (not a
modal, not a fall-through search) showing every angle on a contractor.
Reachable from the Co-Pilot dashboard, the staffers console, and the
search box — all anchor-wrap contractor names to /contractor?name=...

What's new on the page:

1. PROJECT INDEX — build-signal score
   Single 0-100 number with the drivers laid out beneath. Driver list
   is staffer-readable: "59 Chicago permits in 180d (+30) · OSHA 20
   inspections (-25) · federal contractor (+15)". Score weights are
   placeholders to be replaced by an ML model once the 12 awaiting
   sources ship — the current 6 wired signals would not give a real
   model enough features.

2. HEAT MAP — every Chicago permit they've been contact_1 or contact_2
   on, last 24 months, plotted on a leaflet dark map. Color by cost
   (green <$100K, amber $100K-$1M, red ≥$1M), radius proportional to
   cost so the staffer sees where money + activity concentrates. Click
   a marker for permit detail (cost, date, work type, address, permit
   ID). All 50 of Turner Construction's geocoded recent permits in
   Chicago plot end-to-end.

3. ACTIVITY TIMELINE — monthly permit count, bar chart, with the
   first/last month labels so the staffer sees momentum. Tooltip on
   each bar gives the count and total cost for that month.

4. 12 AWAITING SOURCES — placeholder cards for the public datasets
   that would 3× the build-signal feature count. Each card has:
     - source name (real, e.g. DOL Wage & Hour, EPA ECHO, MSHA, BBB)
     - one-liner in coordinator language ("Has this contractor stiffed
       workers? Will they pay our staffing invoices?")
     - "Would show:" sample shape so the engineering scope is concrete
   Order is staffing-decision relevance:
     1. DOL Wage & Hour (WHD violations)
     2. State Licensure Boards (active license + expiry)
     3. Surety Bond Capacity (bonding ceiling)
     4. EPA ECHO Compliance (env violations at sites)
     5. DOT/FMCSA Carrier Safety (crash + OOS rates)
     6. BBB Complaints + Rating
     7. PACER Civil Suits (FLSA / Title VII / ADA)
     8. UCC Lien Filings (cash flow distress)
     9. D&B / Credit Bureau (PAYDEX, payment behavior)
    10. State UI Employer Claims (workforce stability)
    11. MSHA Mine Safety (excavation / aggregate / heavy)
    12. Registered Apprenticeships (DOL RAPIDS pipeline)

Server-side: entity.ts fetchContractorHistory now pulls the 50 most
recent permits with id + lat/lng + work_description, so the heat map
and timeline have what they need without a second SQL hop. The
ContractorHistory.recent_permits type gained the optional fields.

Front-end: contractor.html got 4 new render sections, leaflet wiring
(stylesheet + script in head), placeholder grid CSS, and a PLACEHOLDERS
const at the bottom with the 12 sources. All popup HTML is built via
DOM construction (textContent + appendChild) — no innerHTML, no XSS.

console.html: contractor names from /intelligence/permit_contracts now
anchor-wrapped to /contractor?name=... so the click-through J described
works from the staffers console too. Click stops propagation so the
permit details element doesn't toggle on the same click.

Verified end-to-end via playwright — Turner Construction profile shows:
  PIX score "Mixed signals — review drivers below"
  Heat map: "50 permits plotted · green/amber/red"
  4 section labels in order
  12 placeholder cards in the documented order
2026-04-27 21:28:45 -05:00
root 52d2da2f44 demo: G — per-staffer hot-swap index (synthetic coordinator personas)
Same corpus, different relevance gradient per staffer. Three personas
defined in mcp-server/index.ts STAFFERS roster (Maria/IL, Devon/IN,
Aisha/WI), each with a primary state + secondary cities. Server-side:
/intelligence/chat smart_search accepts a staffer_id body field; when
set, defaults state to the staffer's territory and labels the playbook
context as theirs. The playbook patterns query also defaults its geo
to the staffer's primary city/state, so the recurring-skills/cert
breakdowns reflect what they actually fill, not the global IL prior.

Front-end: a staffer selector dropdown beside the existing state/role
filters. Picking a staffer auto-pins state to their territory, shows
a greeting line, relabels the MEMORY panel as MARIA'S/DEVON'S/AISHA'S
MEMORY, and sends staffer_id to chat for scoping.

Dropdown is populated from /staffers (NOT /api/staffers — the generic
/api/* passthrough sends everything under /api/ to the Rust gateway,
which doesn't own the roster). loadStaffers runs at window-load
independently of loadDay's Promise.all so the dropdown populates even
if simulation/SQL inits error out.

Verified end-to-end via playwright. Same q="forklift operators":
  no staffer  → 509 workers across MI/OH/IA, MEMORY label
  as Devon    → 89 IN-only (Fort Wayne, Terre Haute), DEVON'S MEMORY
  as Aisha    → 16 WI-only (Milwaukee, Madison, Green Bay), AISHA'S MEMORY
As Maria with q="8 production workers near 60607":
  tags: headcount: 8 · zip 60607 → Chicago, IL · role: production · city: Chicago
  20 workers, MARIA'S MEMORY label, top results in Chicago zips

Closes the demo-side build of A-G from the persona plan:
  A. zip → city/state, B. headcount, C. bare-name, D. temporal,
  E. late-worker triage, F. contractor anchor, G. per-staffer index.
2026-04-27 21:16:52 -05:00
root d44ad3af1e demo: P2 — staffer-language routes (zip, headcount, name, late-triage, ingest log)
Built from a playwright run as three personas:
  Maria   — "8 production workers near 60607 by next Friday, prior-fill at this client"
  Devon   — "what came in last night?"
  Aisha   — "Marcus running late site 4422"

Each one previously fell through to smart_search and returned irrelevant
results (geo wrong, headcount ignored, no triage, no temporal). Now:

A. Zip code → city/state lookup. Chicago zips (606xx, 607xx, 608xx)
   resolve to {city: Chicago, state: IL}; 13 metro prefixes covered.
   Maria's "near 60607" now returns Chicago workers, not Dayton/Green Bay.

B. Headcount parser. "8 production workers" / "12 forklift operators" /
   "5 welders" set top_k 1..200, capped 5..25 for SQL+vector LIMIT.
   Allows 0-2 role words between the count and the worker noun so
   "8 production workers" matches as well as "8 workers".

C. Bare-name profile lookup. Single short capitalized phrase
   ("Marcus" / "Sarah Lopez") triggers a profile route. Per-token LIKE
   AND-joined so "Marcus Rivera" matches "Marcus L. Rivera" without
   hardcoding middle initials.

E. Late-worker / no-show triage. Pattern: <Name> (running late|late|
   no show|sick|out today|called out|can't make it) — pulls profile +
   reliability + responsiveness + recent calls, sources 5 same-role
   same-geo backfills sorted by responsiveness, drafts a client SMS
   the coordinator can copy. Front-end renders triage card + Copy SMS
   button + green backfill list.

F. Contractor name preview anchor. The PROJECT INDEX preview line on
   each permit card now wraps contact_1_name and contact_2_name in
   anchors to /contractor?name=... — clicking a contractor finally
   navigates instead of doing nothing. Click handler stops propagation
   so the details element doesn't toggle.

D. Temporal "what came in" route. last night / today / past N hours /
   recent — surfaces datasets from the catalog whose updated_at is
   within the window, samples one row per dataset to detect worker-
   shape, groups by role for worker tables. Schema-agnostic — drop
   any dataset and it shows up. Currently sparse because no fresh
   ingest has happened today; will populate as ingest runs.

Server: /intelligence/chat smart_search route accepts structured
state/role from the search-form dropdowns (P1 from prior commit) and
now ALSO honors b.state, b.role, q.match for headcount + zip + name +
triage patterns BEFORE falling through to NL parsing.

Front-end: doSearch dispatches on response.type and renders triage,
profile, ingest_log, and miss states with type-specific UI. All DOM
construction uses textContent / appendChild — no innerHTML, no XSS.

Verified end-to-end via playwright drive of devop.live/lakehouse:
  Maria  → 8 Chicago Production Workers (60685, 60662, 60634)
           tags: "headcount: 8 · zip 60607 → Chicago, IL · ..."
  Aisha  → Marcus V. Campbell card + draft SMS + 5 Quincy IL backfills
           "I'm dispatching Scott B. Cooper (96% reliability) to cover."
  Devon  → ingest_log surfaces successful_playbooks_live (last 1h)
  Marcus → 5 profiles (Adams Louisville KY, Jenkins Green Bay WI, ...)

Screenshots: /tmp/persona_v2/{01_maria,02_aisha,03_devon,04_marcus}.png

Restart sequence after these edits: pkill -9 -f "mcp-server/index.ts" ;
cd /home/profit/lakehouse ; bun run mcp-server/index.ts. The bun on
:3700 is not systemd-managed (pre-existing convention).
2026-04-27 21:05:40 -05:00
root 89ac6a9b5b demo: P1 — search filter now actually filters by state and role
The Co-Pilot search box read state and role from the dropdowns (#sst, #srl)
but appended them to the message string as ' in '+st. The server's NL
parser then matched the literal preposition "in" against the case-insensitive
regex /\b(IL|IN|...)\b/i and assigned state IN (Indiana) to every search.
Result: typing "forklift in IL" returned Indiana workers. Same for WI, TX,
any state — all silently became Indiana. That was the "cached/generic
response" the legacy staffing client was seeing.

Two prongs:

1. search.html doSearch() now passes structured fields:
     {message, state, role}
   instead of munging into the message text. Dropdown selections bypass
   NL parsing entirely.

2. /intelligence/chat smart_search route accepts those structured fields
   and prefers them over regex archaeology. Falls back to NL parsing only
   when fields aren't provided. Fixed the regex too: the prepositional
   form (?:in|from)\s+(STATE) wins, the standalone form requires uppercase
   (drops /i flag) so the lowercase preposition "in" can no longer match.

Verified live:
- POST /intelligence/chat {"message":"forklift","state":"IL"}
    → 167 IL forklift operators (Galesburg, Joliet, ...)
- POST /intelligence/chat {"message":"forklift","state":"WI","role":"Forklift Operator"}
    → 16 WI Forklift Operators (Milwaukee, Madison, ...)
- POST /intelligence/chat {"message":"forklift in IL"} (NL fallback)
    → 167 IL workers (regex now correctly distinguishes preposition from state code)

Playwright drove the live UI through devop.live/lakehouse and confirmed the
front-end posts the structured body and the result panel renders the right
state. Restart sequence: kill old bun :3700, bun run mcp-server/index.ts.
2026-04-27 20:49:15 -05:00
26 changed files with 479 additions and 4187 deletions
+1 -5
View File
@@ -6,9 +6,5 @@ __pycache__/
*.pyc *.pyc
# Headshot pool — binary face JPGs are fetched by scripts/staffing/fetch_face_pool.py # Headshot pool — binary face JPGs are fetched by scripts/staffing/fetch_face_pool.py
# (synthetic StyleGAN, ~580MB for 1000 faces). Manifest + fetch script are tracked. # (synthetic StyleGAN, ~100MB for 200 faces). Manifest + fetch script are tracked.
data/headshots/face_*.jpg data/headshots/face_*.jpg
data/headshots/_thumbs/
# ComfyUI on-demand generated portraits (per-worker unique). Cached on first
# request; fully regeneratable via /headshots/generate/:key.
data/headshots_gen/
Generated
-1
View File
@@ -48,7 +48,6 @@ version = "0.1.0"
dependencies = [ dependencies = [
"async-trait", "async-trait",
"axum", "axum",
"lru",
"reqwest", "reqwest",
"serde", "serde",
"serde_json", "serde_json",
-239
View File
@@ -1,239 +0,0 @@
# STATE OF PLAY — Lakehouse
**Last verified:** 2026-04-27 ~20:35 CDT
**Verified by:** live probe, not memory.
> **Read this FIRST.** When the user says "we're working on lakehouse," they mean the working code captured below — NOT what `git log` framed as "the cutover" or what memory snapshots from 2 days ago suggest. If memory contradicts this file, this file wins. Update it when something is verified working — not when a phase finishes.
---
## VERIFIED WORKING RIGHT NOW
### The client demo (Staffing Co-Pilot)
**Public URL:** `https://devop.live/lakehouse/` — 200, "Staffing Co-Pilot" (159 KB SPA, leaflet maps, dark theme).
**Local URL:** `http://localhost:3700/` — same page, served by `mcp-server/index.ts` (PID 1271, started 09:48 CDT today).
**The staffers console** (the one the client was thoroughly impressed with):
- `https://devop.live/lakehouse/console` — 200, "Lakehouse — What Your Staffing System Would Do" (26 KB)
- Pulls project index via `/api/catalog/datasets` (36 datasets) + playbook memory via `/api/vectors/playbook_memory/stats` (4,701 entries with embeddings, real ops like *"fill: Maintenance Tech x2 in Milwaukee, WI"*)
Client-visible flow that works end-to-end on the public URL:
| Endpoint | Sample output |
|---|---|
| `GET /api/catalog/datasets` | 36 datasets indexed: timesheets 1M, call_log 800K, workers_500k 500K, email_log 500K, workers_100k 100K, candidates 100K, placements 50K, job_orders 15K, successful_playbooks_live 2,077 |
| `GET /api/vectors/playbook_memory/stats` | 4,701 fill operations with embeddings |
| `GET /system/summary` | 36 datasets, 2.98M rows, 60 indexes, 500K workers loaded, 1K candidates |
| `POST /intelligence/staffing_forecast` | 744 Production Workers needed in 30d, 11,281 bench (4,687 reliable), coverage 1,444%, risk=ok. Same for Electrician (need 32, bench 2,440) and Maintenance Tech (need 17, bench 5,004). |
| `POST /intelligence/permit_contracts` | permit `3442956` $500K → 3 Production Workers, 886-candidate pool, 95% fill, $36K gross. 5 more Chicago permits with 8 workers each, same pool, 95% fill, $96K each. |
| `POST /intelligence/market` | major Chicago permits ranked: $730M O'Hare, $615M 307 N Michigan, $580M casino, $445M Loop transit (real geo coords). |
| `POST /intelligence/permit_entities` | architects + contractors from permit contacts (e.g. "KACPRZYNSKI, ANDY", "SLS ELECTRICAL SERVICE"). |
| `POST /intelligence/activity` + `/intelligence/arch_signals` + `/intelligence/chat` | all 200 |
The demo tells the story: *"upcoming Chicago contracts → workers needed → coverage from the bench → architects/contractors involved → revenue and margin."* That's the "live data + anticipating contracts + complete workflow" pitch — working as of right now.
### Backend, verified live this session
| Surface | State |
|---|---|
| Gateway `:3100` | up, 4 providers configured, `/v1/health` 200 with 500K workers loaded |
| MCP server `:3700` (Co-Pilot demo) | up, all `/intelligence/*` endpoints respond |
| VCP UI `:3950` | started this session, `/data/*` 200, real numbers |
| Observer `:3800` | ring full (2,000/2,000) — older events evicted, query Langfuse for 24h-ago state |
| Sidecar `:3200` | up |
| Langfuse `:3001` | recording, `gw:/log` + `v1.chat:openrouter` traces visible |
| LLM Team UI `:5000` | up, only `extract` mode registered |
| OpenCode fleet | **40 models reachable through one `sk-*` key** (verified live `GET https://opencode.ai/zen/v1/models`) |
OpenCode catalog (live):
- Claude: opus-4-7, opus-4-6, opus-4-5, opus-4-1, sonnet-4-6, sonnet-4-5, sonnet-4, haiku-4-5
- GPT-5: 5.5-pro, 5.5, 5.4-pro, 5.4, 5.4-mini, 5.4-nano, 5.3-codex-spark, 5.3-codex, 5.2, 5.2-codex, 5.1-codex-max, 5.1-codex, 5.1-codex-mini, 5.1, 5-codex, 5-nano, 5
- Gemini: 3.1-pro, 3-flash
- GLM: 5.1, 5
- Minimax: m2.7, m2.5
- Kimi: k2.6, k2.5
- Qwen: 3.6-plus, 3.5-plus
- Other: BIG-PKL (was a typo-prone name in the catalog, model id starts with "big-pkl-something")
- Free tier: minimax-m2.5-free, hy3-preview-free, ling-2.6-flash-free, trinity-large-preview-free
### The substrate (frozen — do not re-architect)
- Distillation v1.0.0 at tag `e7636f2` — **145/145 bun tests pass, 22/22 acceptance, 16/16 audit-full**
- Output: `data/_kb/distilled_{facts,procedures,config_hints}.jsonl` + `data/vectors/distilled_{factual,procedural,config_hint}_v20260423102847.parquet`
- Auditor cross-lineage: Kimi K2.6 ↔ Haiku 4.5 alternation, Opus auto-promote on diffs >100k chars, **per-PR cap=3 with auto-reset on new head SHA**
- Pathway memory: 88 traces, 11/11 successful replays (probation gate crossed)
- Mode runner: 5 native modes; `codereview_isolation` is default; composed-corpus auto-downgrade verified Apr 26 (composed lost 5/5 vs isolation, p=0.031)
### Matrix indexer
30+ live corpora including:
- 5 versions of `workers_500k_v1..v9` (50K embedded chunks each)
- 11 batched 2K-row shards `w500k_b3..b17`
- `chicago_permits_v1` (3,420), `resumes_100k_v2` (100K candidates), `ethereal_workers_v1` (10K)
- `lakehouse_arch_v1` (2,119), `lakehouse_symbols_v1` (2,470), `lakehouse_answers_v1` (1,269), `scrum_findings_v1` (1,260)
- `kb_team_runs_v1` (12,693) + `kb_team_runs_agent` (4,407) — LLM-team play history embedded
- `distilled_factual_v20260423102507` (8) — distillation output
### Code health
- `cargo check --workspace` → **0 warnings, 0 errors**
- `bun test auditor + tests/distillation` → **145/145 pass**
- `ui/server.ts` + `auditor.ts` bundle clean
---
## DO NOT RELITIGATE
- **PR #11 is merged into `origin/main` as `ed57eda`** — do not "still need to merge PR #11."
- **Distillation tag `distillation-v1.0.0` at `e7636f2` is FROZEN** — do not re-architect schemas, scorer rules, audit fixtures.
- **Kimi forensic HOLD verdict (2026-04-27) was 2/8 false + 6/8 latent** — do not re-debate, see `reports/kimi/audit-last-week-full.md`.
- **`candidates_safe` `vertical` column bug** — fixed at catalog metadata layer in commit `c3c9c21`. Do not "discover" it again.
- **Decisions A/B/C/D from `synthetic-data-gap-report.md`** — all four scripts shipped today (`d56f08e`, `940737d`, `c3c9c21`). Do not "ask J for approval."
- **`workers_500k.phone` type fixup** — already string. The fixup script is idempotent; running it is a no-op.
- **`client_workerskjkk` typo dataset** — was breaking every SQL query (catalog had it registered, file didn't exist). Removed via `DELETE /catalog/datasets/by-name/client_workerskjkk` this session. Do not re-add. Adding a startup gate that errors on unrecognized parquet names is the long-term fix per now.md Step 2C.
---
## FIXES MADE THIS SESSION (2026-04-27 evening)
1. **`crates/gateway/src/v1/iterate.rs:93`** — `state` → `_state` (cleared the one cargo warning).
2. **`lakehouse-ui.service` (Dioxus)** — disabled. Was failing 7,242 times against a missing `target/dx/ui/debug/web/public` build dir. `systemctl stop && disable`.
3. **VCP UI on `:3950`** — started `bun run ui/server.ts` (PID 1162212, log `/tmp/lakehouse_ui.log`). `/data/*` endpoints now 200 with real data.
4. **`client_workerskjkk` catalog entry** — `DELETE /catalog/datasets/by-name/client_workerskjkk` removed the dead manifest. **This was the actual root cause** of `/system/summary` reporting `workers_500k_rows: 0` and the demo showing zero bench. Every SQL query was failing schema inference on the missing file before reaching its target table. Fixed → `workers_500k_rows: 500000`, `candidates_rows: 1000`, demo coverage flipped from "critical 0%" to actual percentages on devop.live/lakehouse.
## FIXES MADE THIS SESSION (2026-04-28 early — face pool)
5. **Synthetic StyleGAN face pool — 1000 faces, gender+race+age tagged.** `scripts/staffing/fetch_face_pool.py` fetches from thispersondoesnotexist.com; `scripts/staffing/tag_face_pool.py --min-age 22` runs deepface and excludes minors. `data/headshots/manifest.jsonl` now has gender (494 men / 458 women), race (caucasian 662 · east_asian 128 · hispanic 86 · middle_eastern 59 · black 14 · south_asian 3), age, and 48 minor exclusions. Server pool = 952 servable faces.
6. **`mcp-server/index.ts:1308` `/headshots/:key` route** — gender×race×age intersection bucketing with graceful fallback (gender-only → all). Same key always returns same face; different keys spread evenly.
7. **`/headshots/_thumbs/` pre-resized 384×384 webp** (60× smaller: 587KB → ~11KB). Without this, 40-card grids overran Chrome's parallel-connection budget and ~75% of tiles never finished decoding. Generated via parallel ffmpeg (`xargs -P 8`); `.gitignore`d.
8. **`mcp-server/search.html` + `console.html`** — dropped `img.loading='lazy'`. With 11KB thumbs, eager load is cheap (~500KB for 50 cards) and avoids the off-screen race that lazy decode produced.
9. **ComfyUI on-demand uniqueness — `serve_imagegen.py:32`** added `seed` to `_cache_key()` (was caching by prompt only — 3 different worker seeds collapsed to 1 cached image). Verified: seed=839185194/195/196 → 3 distinct md5s.
10. **`mcp-server/index.ts:1234` `/headshots/generate/:key`** — ComfyUI hot-path that derives a deterministic-per-worker seed via djb2-style hash; cold ~1.5s, cached ~1ms. Worker prompt format: `professional corporate headshot portrait of a {age}-year-old {race} {gender}, {role}, neutral expression, plain studio background, soft natural lighting, sharp focus, photorealistic, dslr`. Cache at `data/headshots_gen/` (gitignored, regeneratable).
11. **Confidence-default name resolution** in `search.html` — `genderFor()` and `guessEthnicityFromFirstName()` lookup tables (FEMALE_NAMES, MALE_NAMES, NAMES_HISPANIC, NAMES_BLACK, NAMES_SOUTH_ASIAN, NAMES_EAST_ASIAN, NAMES_MIDDLE_EASTERN). Xavier → man+hispanic, Aisha → woman+black, etc. Every worker resolves to a face-pool bucket.
End-to-end verified: playwright run on `https://devop.live/lakehouse/?q=forklift+operators+IL` → 21/21 cards loaded, 0 broken, all 384×384 webp thumbs.
---
## OPEN — but not blocking the demo
| Item | What | When to act |
|---|---|---|
| `modes.toml` `staffing_inference.matrix_corpus` | still says `workers_500k_v8`. v9 in vector index is from Apr 17 (raw-sourced, not safe-view). The new `build_workers_v9.sh` rebuilds from `workers_safe`. | Run when you have 30+ min for the rebuild. |
| Open PRs #6, #7, #10 | sitting since Apr 22-24, auditor verdicts on disk at `data/_auditor/kimi_verdicts/{6,7,10}-*.json` | Read verdicts, decide reconcile/close. |
| `test/enrich-prd-pipeline` branch | 35 unmerged commits, includes more-evolved auditor/inference.ts (666 vs main's 580 lines), curation+fact-extractor wiring | Reconcile or formally archive — see `memory/project_unmerged_architecture_work.md`. |
| `federation-hnsw-trials` stash | Lance + S3/MinIO prototype, `aws-config` crate added, 708 insertions | Phase B from EXECUTION_PLAN.md — revisit when Parquet vector ceiling actually hurts. |
| `candidates` manifest drift | manifest 100K vs SQL 1K. Cosmetic. | Run a metadata resync if it matters. |
---
## RUNTIME CHEATSHEET
```bash
# Verify the demo (public + local both work)
curl -sS https://devop.live/lakehouse/ # Co-Pilot HTML
curl -sS https://devop.live/lakehouse/console # staffers console
curl -sS -X POST https://devop.live/lakehouse/intelligence/staffing_forecast \
-d '{}' -H 'content-type: application/json' \
| jq '.forecast[] | {role, demand_workers, bench_total, coverage_pct, risk}'
# Restart sequence (after Rust changes)
sudo systemctl restart lakehouse.service # gateway :3100
sudo systemctl restart lakehouse-auditor # auditor daemon
sudo systemctl restart lakehouse-observer # observer :3800
# UI bun on :3950 is NOT systemd-managed (lakehouse-ui.service is disabled).
# Restart manually: kill <pid>; nohup bun run ui/server.ts > /tmp/lakehouse_ui.log 2>&1 &
# Health checks
curl -sS http://localhost:3100/v1/health | jq # workers_count, providers
curl -sS http://localhost:3100/vectors/pathway/stats | jq
curl -sS http://localhost:3100/v1/usage | jq # since-restart cost
curl -sS http://localhost:3700/system/summary | jq # dataset counts
```
---
## VISION — what we're actually building (not what's done)
J's framing for the legacy staffing company:
- Pull live data, anticipate contracts based on Chicago permits → real architect/contractor associations, headcount, time period, money, scope.
- Hybrid + memory index → search large corpora cheaply.
- Email comes in → verify against contract; SMS comes in → alert when index changes.
- Real-time.
- Invent metrics nobody else has using the hybrid index.
- Next stage: workers download an app → geolocation clock-in → automatic responsiveness measurement, no user effort, with incentives for using it.
- Find people getting certificates (passive cert tracking).
- Pull union data → bring contracts that work for **employees**, not just employers.
- All metrics visible, nothing hidden, value-aligned with what each side actually needs.
If a future session is shaving away from this vision toward "fix the cutover" or "land Phase X," the vision wins. Phases are scaffolding for the vision, not the goal.
---
## CURRENT PLAN — fix the demo for the legacy staffing client
Built from playwright audit of the live demo (2026-04-27 evening). Each item ends in something the client can SEE, not internal cleanups.
**Demo state is anchored by git tag `demo-2026-04-27`** (commit `ed57eda`, the merge of PR #11). To restore code state: `git checkout demo-2026-04-27`. To restore runtime state: `DELETE /catalog/datasets/by-name/client_workerskjkk` (catalog hot-fix is not in git).
### P1 — Search box that actually filters (highest visible impact)
**Problem:** typing in `#sq` and pressing Enter fires `POST /intelligence/chat` with body `{"message":"<query>"}`. The state (`#sst`) and role (`#srl`) selects are ignored — never sent in the body. So every search returns a generic chat completion, never a SQL+vector hybrid filter against `workers_500k`. That is the "cached/generic response" the client sees.
**Fix:** in `mcp-server/search.html`, change the search-submit handler to call the real worker search endpoint with `{query, state, role, top_k}`. The MCP `search_workers` tool surface already exists; route the form there. Render returned worker rows in the existing card grid.
**Done when:** typing "forklift" + state IL + role "Forklift Operator" returns ≤ top_k IL Forklift Operators, and changing state to WI returns different workers.
### P2 — Contractor-name click → `/contractor` profile page
**Problem:** clicking a contractor name in any rendered card stays on `/lakehouse/`. URL doesn't change.
**Fix:** wrap contractor names in `<a href="/contractor?name=<encoded>">`. The page `mcp-server/contractor.html` (14.8 KB, "Contractor Profile · Staffing Co-Pilot") already exists at `/contractor` and the data endpoint `/intelligence/contractor_profile` already returns rich data.
**Then check contractor.html actually shows:** full history of every record the database has on that contractor + heat map of locations underneath + relevant info (per J 2026-04-27). If the page is incomplete, finish it. Otherwise just wire the link.
**Done when:** clicking "KACPRZYNSKI, ANDY" opens a profile with: every Chicago permit they're contact_1 or contact_2 on, a leaflet map with markers for each address, and any matched workers from prior placements at their sites.
### P3 — Substrate signal at the bottom shows the right numbers
**Problem:** J reports the bottom panel says "playbook memory empty, 80 traces 0 replies." Reality from the live endpoints: `/api/vectors/playbook_memory/stats` = 4,701 entries with embeddings; `/vectors/pathway/stats` = 88 traces, 11/11 replays.
**Fix:** find the renderer in search.html that builds the substrate signal panel; verify it's hitting the right endpoints and reading the right keys; fix shape mismatches.
**Done when:** bottom panel shows real numbers (4,701 playbooks, 88 traces, 11/11 replays) and references at least one specific recent operation from the playbook stats sample.
### P4 — Top nav reflects today's architecture
**Problem:** Walkthrough/Architecture/Spec/Onboard/Alerts/Workspaces tabs all return 200 but content is from old architecture. Doesn't mention: gateway scratchpad, memory indexer, ranker, mode runner, OpenCode 40-model fleet, distillation substrate, auditor cross-lineage.
**Fix:** rewrite `mcp-server/proof.html` (or add a single new page "What's running" that replaces Architecture+Spec) to describe what's actually shipped as of `demo-2026-04-27`. Keep one architecture page, drop redundancy. Either complete or hide Onboard/Alerts/Workspaces — J's call which.
**Done when:** the architecture page tells a non-technical reader, in 2 minutes, what each piece does in coordinator-relatable terms ("intern that read every email", not "3-stage adversarial inference pipeline").
### P5 — Caching for the project-index build_signal (J flagged unfinished)
**Problem:** "we never finished our caching for project index build signal it's not pulling new information." Need to find what `build_signal` refers to. Likely a scrum/auditor signal that should rebuild the `lakehouse_arch_v1` corpus on commit but isn't wired to.
**Fix:** identify the build-signal pipeline (likely in `auditor/` or `crates/vectord/`), wire its emit to a corpus rebuild, verify by making a test commit and watching the new chunk appear in `/vectors/indexes` for `lakehouse_arch_v1`.
**Done when:** committing a new file to `crates/` causes `lakehouse_arch_v1` chunk_count to increase within N minutes.
### P0 — Anchor the demo state (DONE)
Tagged `ed57eda` as `demo-2026-04-27`. Future sessions: `git checkout demo-2026-04-27` to land in this exact code state.
---
## EXECUTION ORDER
1. **P1 first** — biggest visible bug, ~30-60 min
2. **P2 next** — contractor click is the second-biggest "doesn't work" the client sees, ~20 min if profile is mostly done
3. **P3** — small fix, big "looks alive" win
4. **P4** — biggest scope; might split across sessions
5. **P5** — feature work, only after the visible bugs are fixed
Each item commits independently with the format `demo: P<n> — <one-line>` so the commit log doubles as a progress journal. After each merge to main, re-tag `demo-latest` to point at the new HEAD.
Stop here and let J pick which item to start with. Do not silently extend scope.
+3 -5
View File
@@ -16,14 +16,12 @@ import type { Gap, Proposal } from "./types.ts";
// Phase 44 migration (2026-04-27): bot/propose.ts now flows through // Phase 44 migration (2026-04-27): bot/propose.ts now flows through
// the gateway's /v1/chat instead of hitting the sidecar's /generate // the gateway's /v1/chat instead of hitting the sidecar's /generate
// directly. /v1/usage tracks the call, Langfuse traces it, observer // directly. /v1/usage tracks the call, Langfuse traces it, observer
// sees it. Gateway owns the routing. // sees it. Same upstream model (CLOUD_MODEL gpt-oss:120b on
// // Ollama Cloud) — gateway just owns the routing.
// 2026-04-28: gpt-oss:120b → deepseek-v3.2 via Ollama Pro. Newer
// DeepSeek revision, faster, still on the same OLLAMA_CLOUD_KEY.
const GATEWAY_URL = process.env.LH_GATEWAY_URL ?? "http://localhost:3100"; const GATEWAY_URL = process.env.LH_GATEWAY_URL ?? "http://localhost:3100";
const REPO_ROOT = "/home/profit/lakehouse"; const REPO_ROOT = "/home/profit/lakehouse";
const PRD_PATH = `${REPO_ROOT}/docs/PRD.md`; const PRD_PATH = `${REPO_ROOT}/docs/PRD.md`;
const CLOUD_MODEL = process.env.LH_BOT_MODEL ?? "deepseek-v3.2"; const CLOUD_MODEL = process.env.LH_BOT_MODEL ?? "gpt-oss:120b";
const MAX_TOKENS = 6000; const MAX_TOKENS = 6000;
export async function findGaps(): Promise<Gap[]> { export async function findGaps(): Promise<Gap[]> {
+2 -7
View File
@@ -44,10 +44,7 @@ name = "staffing_inference"
# pattern generalizes beyond code review. # pattern generalizes beyond code review.
preferred_mode = "staffing_inference_lakehouse" preferred_mode = "staffing_inference_lakehouse"
fallback_modes = ["ladder", "consensus", "pipeline"] fallback_modes = ["ladder", "consensus", "pipeline"]
# 2026-04-28: gpt-oss-120b:free → kimi-k2.6 via Ollama Pro. Coding- default_model = "openai/gpt-oss-120b:free"
# specialized, faster than gpt-oss, on the same OLLAMA_CLOUD_KEY so
# no extra provider hop.
default_model = "kimi-k2.6"
matrix_corpus = "workers_500k_v8" matrix_corpus = "workers_500k_v8"
[[task_class]] [[task_class]]
@@ -61,9 +58,7 @@ matrix_corpus = "kb_team_runs_v1"
name = "doc_drift_check" name = "doc_drift_check"
preferred_mode = "drift" preferred_mode = "drift"
fallback_modes = ["validator"] fallback_modes = ["validator"]
# 2026-04-28: gpt-oss:120b → gemini-3-flash-preview via Ollama Pro. default_model = "gpt-oss:120b"
# Speed leader on factual checking, same OLLAMA_CLOUD_KEY.
default_model = "gemini-3-flash-preview"
matrix_corpus = "distilled_factual_v20260423095819" matrix_corpus = "distilled_factual_v20260423095819"
[[task_class]] [[task_class]]
+5 -10
View File
@@ -27,15 +27,10 @@ name = "ollama_cloud"
base_url = "https://ollama.com" base_url = "https://ollama.com"
auth = "bearer" auth = "bearer"
auth_env = "OLLAMA_CLOUD_KEY" auth_env = "OLLAMA_CLOUD_KEY"
default_model = "deepseek-v3.2" default_model = "gpt-oss:120b"
# Cloud-tier Ollama (Pro plan as of 2026-04-28). Key resolved from # Cloud-tier Ollama. Key resolved from OLLAMA_CLOUD_KEY env at gateway
# OLLAMA_CLOUD_KEY at gateway boot; Pro tier upgraded the account so # boot. Model-prefix routing: "cloud/<model>" auto-routes here
# rate limits + model access widen without a key change. Model-prefix # (see gateway::v1::resolve_provider).
# routing: "cloud/<model>" auto-routes here. 39-model fleet now
# includes deepseek-v3.2, deepseek-v4-{flash,pro}, gemini-3-flash-
# preview, glm-{5,5.1}, kimi-k2.6, qwen3-coder-next.
# 2026-04-28: default upgraded gpt-oss:120b → deepseek-v3.2 (newest
# DeepSeek revision; kimi-k2:1t still upstream-broken with HTTP 500).
[[provider]] [[provider]]
name = "openrouter" name = "openrouter"
@@ -43,7 +38,7 @@ base_url = "https://openrouter.ai/api/v1"
auth = "bearer" auth = "bearer"
auth_env = "OPENROUTER_API_KEY" auth_env = "OPENROUTER_API_KEY"
auth_fallback_files = ["/home/profit/.env", "/root/llm_team_config.json"] auth_fallback_files = ["/home/profit/.env", "/root/llm_team_config.json"]
default_model = "x-ai/grok-4.1-fast" default_model = "openai/gpt-oss-120b:free"
# Multi-provider gateway. Covers Anthropic, Google, OpenAI, MiniMax, # Multi-provider gateway. Covers Anthropic, Google, OpenAI, MiniMax,
# Qwen, Gemma, etc. Key resolved via crates/gateway/src/v1/openrouter.rs # Qwen, Gemma, etc. Key resolved via crates/gateway/src/v1/openrouter.rs
# resolve_openrouter_key() — env first, then fallback files. # resolve_openrouter_key() — env first, then fallback files.
-1
View File
@@ -12,4 +12,3 @@ serde_json = { workspace = true }
tracing = { workspace = true } tracing = { workspace = true }
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] } reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
async-trait = "0.1" async-trait = "0.1"
lru = "0.12"
+1 -163
View File
@@ -1,10 +1,5 @@
use lru::LruCache;
use reqwest::Client; use reqwest::Client;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use std::num::NonZeroUsize;
use std::sync::Mutex;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
use std::time::Duration; use std::time::Duration;
/// HTTP client for the Python AI sidecar. /// HTTP client for the Python AI sidecar.
@@ -23,42 +18,11 @@ use std::time::Duration;
/// callers that want observability (vectord modules); it's left /// callers that want observability (vectord modules); it's left
/// unset by callers that ARE the gateway internals (avoids self-loops /// unset by callers that ARE the gateway internals (avoids self-loops
/// + redundant hops). /// + redundant hops).
/// Per-text embed cache key. We key on (model, text) so different
/// model selections produce distinct cache lines — a query embedded
/// under nomic-embed-text-v2-moe must NOT collide with the same
/// query under nomic-embed-text v1.
#[derive(Eq, PartialEq, Hash, Clone)]
struct EmbedCacheKey {
model: String,
text: String,
}
/// Default LRU cache size — 4096 entries × ~6KB per 768-d f64
/// vector ≈ 24MB. Sized for typical staffing-domain repetition
/// (coordinator workflows have query repetition rates around 70-90%
/// per session). Tunable via [aibridge].embed_cache_size in the
/// config; 0 disables the cache entirely.
const DEFAULT_EMBED_CACHE_SIZE: usize = 4096;
#[derive(Clone)] #[derive(Clone)]
pub struct AiClient { pub struct AiClient {
client: Client, client: Client,
base_url: String, base_url: String,
gateway_url: Option<String>, gateway_url: Option<String>,
/// Closes the 63× perf gap with Go side. Mirrors the shape of
/// Go's internal/embed/cached.go::CachedProvider — same
/// (model, text) → vector caching, same nil-disable semantics.
/// None = caching disabled (cache_size=0); Some = bounded LRU.
embed_cache: Option<Arc<Mutex<LruCache<EmbedCacheKey, Vec<f64>>>>>,
/// Hit / miss counters for /admin observability + load-test
/// validation. Atomic so Clone'd AiClients share the same counts.
embed_cache_hits: Arc<AtomicU64>,
embed_cache_misses: Arc<AtomicU64>,
/// Pinned at construction time so the EmbedResponse can carry
/// dimension consistently even when every text was a cache hit
/// (no fresh sidecar call to learn the dim from). Set on first
/// successful sidecar embed; checked on every cache hit.
cached_dim: Arc<AtomicU64>,
} }
// -- Request/Response types -- // -- Request/Response types --
@@ -131,49 +95,17 @@ pub struct RerankResponse {
impl AiClient { impl AiClient {
pub fn new(base_url: &str) -> Self { pub fn new(base_url: &str) -> Self {
Self::with_embed_cache(base_url, DEFAULT_EMBED_CACHE_SIZE)
}
/// Constructs an AiClient with an explicit embed-cache size.
/// Pass 0 to disable the cache entirely (matches Go-side
/// CachedProvider's nil-cache semantics).
pub fn with_embed_cache(base_url: &str, cache_size: usize) -> Self {
let client = Client::builder() let client = Client::builder()
.timeout(Duration::from_secs(120)) .timeout(Duration::from_secs(120))
.build() .build()
.expect("failed to build HTTP client"); .expect("failed to build HTTP client");
let embed_cache = if cache_size > 0 {
// SAFETY: cache_size > 0 just verified, NonZeroUsize::new
// returns Some.
let cap = NonZeroUsize::new(cache_size).expect("cache_size > 0");
Some(Arc::new(Mutex::new(LruCache::new(cap))))
} else {
None
};
Self { Self {
client, client,
base_url: base_url.trim_end_matches('/').to_string(), base_url: base_url.trim_end_matches('/').to_string(),
gateway_url: None, gateway_url: None,
embed_cache,
embed_cache_hits: Arc::new(AtomicU64::new(0)),
embed_cache_misses: Arc::new(AtomicU64::new(0)),
cached_dim: Arc::new(AtomicU64::new(0)),
} }
} }
/// Cache hit/miss/size snapshot. Useful for /admin endpoints +
/// load-test validation ("did the cache fire as expected?").
pub fn embed_cache_stats(&self) -> (u64, u64, usize) {
let hits = self.embed_cache_hits.load(Ordering::Relaxed);
let misses = self.embed_cache_misses.load(Ordering::Relaxed);
let len = self
.embed_cache
.as_ref()
.map(|c| c.lock().map(|g| g.len()).unwrap_or(0))
.unwrap_or(0);
(hits, misses, len)
}
/// Same as `new`, but every `generate()` is routed through /// Same as `new`, but every `generate()` is routed through
/// `${gateway_url}/v1/chat` (provider=ollama) for observability. /// `${gateway_url}/v1/chat` (provider=ollama) for observability.
/// Use this for callers OUTSIDE the gateway. Inside the gateway /// Use this for callers OUTSIDE the gateway. Inside the gateway
@@ -195,104 +127,10 @@ impl AiClient {
resp.json().await.map_err(|e| format!("invalid response: {e}")) resp.json().await.map_err(|e| format!("invalid response: {e}"))
} }
/// Embed with per-text LRU caching. Mirrors Go-side
/// CachedProvider behavior: cache key is (model, text);
/// cache-hit texts skip the sidecar; cache-miss texts batch
/// into a single sidecar call; results are interleaved in the
/// caller's input order.
///
/// Closes ~95% of the load-test perf gap vs Go side (loadgen
/// 2026-05-01: Rust 128 RPS → with cache ≥ 7000 RPS expected
/// for warm-cache workloads). Cold-cache behavior unchanged
/// (every text is a miss → single sidecar call, identical to
/// pre-cache).
pub async fn embed(&self, req: EmbedRequest) -> Result<EmbedResponse, String> { pub async fn embed(&self, req: EmbedRequest) -> Result<EmbedResponse, String> {
let model_key = req.model.clone().unwrap_or_default();
// Fast path: cache disabled → original behavior.
let Some(cache) = self.embed_cache.as_ref() else {
return self.embed_uncached(&req).await;
};
if req.texts.is_empty() {
return self.embed_uncached(&req).await;
}
// First pass: check cache for each text. Track which positions
// need a sidecar fetch.
let mut embeddings: Vec<Option<Vec<f64>>> = vec![None; req.texts.len()];
let mut miss_indices: Vec<usize> = Vec::new();
let mut miss_texts: Vec<String> = Vec::new();
{
let mut guard = cache.lock().map_err(|e| format!("cache lock poisoned: {e}"))?;
for (i, text) in req.texts.iter().enumerate() {
let key = EmbedCacheKey { model: model_key.clone(), text: text.clone() };
if let Some(vec) = guard.get(&key) {
embeddings[i] = Some(vec.clone());
self.embed_cache_hits.fetch_add(1, Ordering::Relaxed);
} else {
miss_indices.push(i);
miss_texts.push(text.clone());
self.embed_cache_misses.fetch_add(1, Ordering::Relaxed);
}
}
}
// All hit? Return immediately. Use cached_dim to populate
// the response dimension (no sidecar to ask).
if miss_indices.is_empty() {
let dim = self.cached_dim.load(Ordering::Relaxed) as usize;
let dim = if dim == 0 { embeddings[0].as_ref().map(|v| v.len()).unwrap_or(0) } else { dim };
return Ok(EmbedResponse {
embeddings: embeddings.into_iter().map(|opt| opt.expect("filled")).collect(),
model: req.model.unwrap_or_else(|| "nomic-embed-text".to_string()),
dimensions: dim,
});
}
// Second pass: fetch the misses in one sidecar call.
let miss_req = EmbedRequest { texts: miss_texts.clone(), model: req.model.clone() };
let resp = self.embed_uncached(&miss_req).await?;
if resp.embeddings.len() != miss_texts.len() {
return Err(format!(
"embed cache: sidecar returned {} embeddings for {} texts",
resp.embeddings.len(),
miss_texts.len()
));
}
// Pin cached_dim on first successful response.
if resp.dimensions > 0 {
self.cached_dim.store(resp.dimensions as u64, Ordering::Relaxed);
}
// Insert misses into cache + fill response slots.
{
let mut guard = cache.lock().map_err(|e| format!("cache lock poisoned: {e}"))?;
for (j, idx) in miss_indices.iter().enumerate() {
let key = EmbedCacheKey {
model: model_key.clone(),
text: miss_texts[j].clone(),
};
let vec = resp.embeddings[j].clone();
guard.put(key, vec.clone());
embeddings[*idx] = Some(vec);
}
}
Ok(EmbedResponse {
embeddings: embeddings.into_iter().map(|opt| opt.expect("filled")).collect(),
model: resp.model,
dimensions: resp.dimensions,
})
}
/// Direct sidecar call — original pre-cache behavior. Used
/// internally by embed() for cache-miss batches and as the
/// transparent fallback when the cache is disabled.
async fn embed_uncached(&self, req: &EmbedRequest) -> Result<EmbedResponse, String> {
let resp = self.client let resp = self.client
.post(format!("{}/embed", self.base_url)) .post(format!("{}/embed", self.base_url))
.json(req) .json(&req)
.send() .send()
.await .await
.map_err(|e| format!("embed request failed: {e}"))?; .map_err(|e| format!("embed request failed: {e}"))?;
+15 -24
View File
@@ -582,10 +582,10 @@ impl ExecutionLoop {
/// Phase 20 step (8) — T3 overseer escalation. /// Phase 20 step (8) — T3 overseer escalation.
/// ///
/// When the local executor/reviewer loop can't self-correct, call /// When the local executor/reviewer loop can't self-correct, call
/// the cloud overseer (`claude-opus-4-7` via OpenCode Zen) with /// the cloud overseer (`gpt-oss:120b` via Ollama Cloud) with (a)
/// (a) the KB context — recent outcomes + prior corrections for /// the KB context — recent outcomes + prior corrections for this
/// this sig_hash + task_class, across every profile that has run /// sig_hash + task_class, across every profile that has run it —
/// it — and (b) the recent log tail. Its output is appended as a /// and (b) the recent log tail. Its output is appended as a
/// `system` role turn so the next executor generation sees it, /// `system` role turn so the next executor generation sees it,
/// AND written to `data/_kb/overseer_corrections.jsonl` so every /// AND written to `data/_kb/overseer_corrections.jsonl` so every
/// future profile activation reads from the same learning pool. /// future profile activation reads from the same learning pool.
@@ -593,16 +593,9 @@ impl ExecutionLoop {
/// This is the "pipe to the overviewer" piece from 2026-04-23 — /// This is the "pipe to the overviewer" piece from 2026-04-23 —
/// the overseer is now a first-class KB consumer AND producer, not /// the overseer is now a first-class KB consumer AND producer, not
/// a one-shot correction oracle. /// a one-shot correction oracle.
///
/// 2026-04-28: routed through OpenCode (Zen tier) for Claude Opus
/// 4.7. Frontier reasoning matters here because the overseer fires
/// only after local self-correction has failed twice — by that
/// point we need the strongest reasoning available, not the
/// cheapest token. Frequency is low so the Zen pay-per-token cost
/// stays bounded.
async fn escalate_to_overseer(&mut self, turn: u32, reason: &str) -> Result<(), String> { async fn escalate_to_overseer(&mut self, turn: u32, reason: &str) -> Result<(), String> {
let Some(opencode_key) = self.state.opencode_key.clone() else { let Some(cloud_key) = self.state.ollama_cloud_key.clone() else {
return Err("OPENCODE_API_KEY not configured — skipping escalation".into()); return Err("OLLAMA_CLOUD_KEY not configured — skipping escalation".into());
}; };
let kb = KbContext::load_for(&sig_hash(&self.req), &self.req.task_class).await; let kb = KbContext::load_for(&sig_hash(&self.req), &self.req.task_class).await;
@@ -611,18 +604,16 @@ impl ExecutionLoop {
let started = std::time::Instant::now(); let started = std::time::Instant::now();
let start_time = chrono::Utc::now(); let start_time = chrono::Utc::now();
let chat_req = crate::v1::ChatRequest { let chat_req = crate::v1::ChatRequest {
model: "claude-opus-4-7".to_string(), model: "gpt-oss:120b".to_string(),
messages: vec![crate::v1::Message::new_text("user", prompt.clone())], messages: vec![crate::v1::Message::new_text("user", prompt.clone())],
temperature: Some(0.1), temperature: Some(0.1),
max_tokens: None, max_tokens: None,
stream: Some(false), stream: Some(false),
// Anthropic models on opencode reject `think` (handled in think: Some(true), // overseer KEEPS thinking (Phase 20 rule)
// the adapter), but we keep the intent flag for parity. provider: Some("ollama_cloud".into()),
think: Some(true),
provider: Some("opencode".into()),
}; };
let resp = crate::v1::opencode::chat(&opencode_key, &chat_req).await let resp = crate::v1::ollama_cloud::chat(&cloud_key, &chat_req).await
.map_err(|e| format!("opencode: {e}"))?; .map_err(|e| format!("ollama_cloud: {e}"))?;
let latency_ms = started.elapsed().as_millis() as u64; let latency_ms = started.elapsed().as_millis() as u64;
let end_time = chrono::Utc::now(); let end_time = chrono::Utc::now();
let correction_text: String = resp.choices.into_iter().next() let correction_text: String = resp.choices.into_iter().next()
@@ -642,8 +633,8 @@ impl ExecutionLoop {
if let Some(lf) = &self.state.langfuse { if let Some(lf) = &self.state.langfuse {
use crate::v1::langfuse_trace::ChatTrace; use crate::v1::langfuse_trace::ChatTrace;
lf.emit_chat(ChatTrace { lf.emit_chat(ChatTrace {
provider: "opencode".into(), provider: "ollama_cloud".into(),
model: "claude-opus-4-7".into(), model: "gpt-oss:120b".into(),
input: vec![crate::v1::Message::new_text("user", prompt.clone())], input: vec![crate::v1::Message::new_text("user", prompt.clone())],
output: correction_text.clone(), output: correction_text.clone(),
prompt_tokens: resp.usage.prompt_tokens, prompt_tokens: resp.usage.prompt_tokens,
@@ -659,7 +650,7 @@ impl ExecutionLoop {
// Append to the transcript so the next executor turn sees it. // Append to the transcript so the next executor turn sees it.
self.append(LogEntry::new( self.append(LogEntry::new(
turn, "system", "claude-opus-4-7", "overseer_correction", turn, "system", "gpt-oss:120b", "overseer_correction",
serde_json::json!({ serde_json::json!({
"reason": reason, "reason": reason,
"correction": correction_text, "correction": correction_text,
@@ -681,7 +672,7 @@ impl ExecutionLoop {
"task_class": self.req.task_class, "task_class": self.req.task_class,
"operation": self.req.operation, "operation": self.req.operation,
"reason": reason, "reason": reason,
"model": "claude-opus-4-7", "model": "gpt-oss:120b",
"correction": correction_text, "correction": correction_text,
"applied_at_turn": turn, "applied_at_turn": turn,
"kb_context_used": kb, "kb_context_used": kb,
+2 -6
View File
@@ -163,11 +163,7 @@ pub async fn query(
// production caller of the Phase 21 primitives — see audit finding // production caller of the Phase 21 primitives — see audit finding
// "Phase 21 Rust primitives are wired but not CALLED by any // "Phase 21 Rust primitives are wired but not CALLED by any
// production surface" from 2026-04-21. // production surface" from 2026-04-21.
// 2026-04-30 model bump: qwen2.5:latest → qwen3.5:latest to match let mut cont_opts = ContinuableOpts::new("qwen2.5:latest");
// the small-model-pipeline local-tier default. Same JSON-clean
// property, more capacity. think=Some(false) preserved — RAG hot
// path doesn't need reasoning traces; direct answers only.
let mut cont_opts = ContinuableOpts::new("qwen3.5:latest");
cont_opts.max_tokens = Some(512); cont_opts.max_tokens = Some(512);
cont_opts.temperature = Some(0.2); cont_opts.temperature = Some(0.2);
cont_opts.shape = ResponseShape::Text; cont_opts.shape = ResponseShape::Text;
@@ -180,7 +176,7 @@ pub async fn query(
// echoes whatever Ollama loaded). Use the configured tier model // echoes whatever Ollama loaded). Use the configured tier model
// for now; if RAG needs to report the actual resolved model, // for now; if RAG needs to report the actual resolved model,
// the runner can add a post-call ps probe later. // the runner can add a post-call ps probe later.
model: "qwen3.5:latest".to_string(), model: "qwen2.5:latest".to_string(),
sources: results, sources: results,
tokens_generated: None, tokens_generated: None,
}) })
File diff suppressed because it is too large Load Diff
+4 -11
View File
@@ -48,13 +48,8 @@ url = "http://localhost:3200"
[ai] [ai]
embed_model = "nomic-embed-text" embed_model = "nomic-embed-text"
# Local-tier defaults bumped 2026-04-30: qwen3.5:latest is the gen_model = "qwen2.5"
# stronger local rung in the 5-loop substrate (per rerank_model = "qwen2.5"
# project_small_model_pipeline_vision.md). Same JSON-clean property
# as qwen2.5, more capacity. Ollama still serves both — bump back
# in this file if a workload regressed.
gen_model = "qwen3.5:latest"
rerank_model = "qwen3.5:latest"
[auth] [auth]
enabled = false enabled = false
@@ -77,9 +72,7 @@ min_recall = 0.9 # never promote below this
max_trials_per_hour = 20 # hard budget cap max_trials_per_hour = 20 # hard budget cap
# Model roster — available for profile hot-swap # Model roster — available for profile hot-swap
# qwen3.5:latest: stronger local rung — JSON-clean, 8K+ context,
# default for gen_model and rerank_model
# qwen3: 8.2B, 40K context, thinking+tools, best for reasoning tasks # qwen3: 8.2B, 40K context, thinking+tools, best for reasoning tasks
# qwen2.5: 7B, 8K context, fast — kept loaded for the 2026-04 era # qwen2.5: 7B, 8K context, fast, good for SQL generation
# comparison runs; new defaults use qwen3.5:latest # mistral: 7B, 8K context, good for general generation
# nomic-embed-text: 137M, embedding-only, used by all profiles # nomic-embed-text: 137M, embedding-only, used by all profiles
+16 -39
View File
@@ -56,12 +56,7 @@ details .body{padding-top:10px;font-size:12px;color:#8b949e}
.worker{display:flex;align-items:center;gap:10px;padding:8px 10px;background:#161b22;border-radius:6px;margin-bottom:4px;font-size:12px;border-left:3px solid #30363d} .worker{display:flex;align-items:center;gap:10px;padding:8px 10px;background:#161b22;border-radius:6px;margin-bottom:4px;font-size:12px;border-left:3px solid #30363d}
.worker .av{width:32px;height:32px;border-radius:50%;background:#0d1117;border:1px solid #21262d;display:flex;align-items:center;justify-content:center;font-weight:600;color:#c9d1d9;font-size:11px;flex-shrink:0;letter-spacing:0.5px;overflow:hidden;position:relative} .worker .av{width:32px;height:32px;border-radius:50%;background:#0d1117;border:1px solid #21262d;display:flex;align-items:center;justify-content:center;font-weight:600;color:#c9d1d9;font-size:11px;flex-shrink:0;letter-spacing:0.5px;overflow:hidden;position:relative}
.worker .av img{position:absolute;inset:0;width:100%;height:100%;object-fit:cover;display:block; .worker .av img{position:absolute;inset:0;width:100%;height:100%;object-fit:cover;display:block}
/* Softening — mirror of search.html. Pulls saturation + contrast off
the SDXL Turbo over-render so faces feel less "AI-generated".
If you tweak one, tweak the other. */
filter: saturate(0.86) contrast(0.93) brightness(1.02) blur(0.3px);
}
.worker[data-role-band="warehouse"]{border-left-color:#58a6ff} .worker[data-role-band="warehouse"]{border-left-color:#58a6ff}
.worker[data-role-band="production"]{border-left-color:#d29922} .worker[data-role-band="production"]{border-left-color:#d29922}
.worker[data-role-band="trades"]{border-left-color:#bc8cff} .worker[data-role-band="trades"]{border-left-color:#bc8cff}
@@ -267,37 +262,6 @@ var NAMES_EAST_ASIAN_C=new Set(['Wei','Mei','Yi','Jin','Chen','Lin','Liu','Wang'
var NAMES_HISPANIC_C=new Set(['Carmen','Carlos','Maria','Diego','Hector','Jorge','Julio','Manuel','Miguel','Pedro','Raul','Ricardo','Roberto','Sergio','Antonio','Esperanza','Luz','Sofia','Lucia','Isabella','Camila','Valentina','Mariana','Elena','Rosa','Catalina','Esteban','Fernando','Eduardo','Javier','Alejandro','Andres','Mateo','Santiago','Sebastian','Emilio','Tomas','Cristina','Daniela','Gabriela','Ximena','Adriana','Beatriz','Pilar','Mercedes','Xavier','Marisol','Guadalupe','Lupita','Inez','Itzel','Yesenia','Joaquin','Ignacio','Rafael','Salvador','Cesar','Arturo','Armando','Hugo','Marco','Alejandra','Felipe','Gerardo','Jaime','Leonardo','Luis','Pablo','Ramon']); var NAMES_HISPANIC_C=new Set(['Carmen','Carlos','Maria','Diego','Hector','Jorge','Julio','Manuel','Miguel','Pedro','Raul','Ricardo','Roberto','Sergio','Antonio','Esperanza','Luz','Sofia','Lucia','Isabella','Camila','Valentina','Mariana','Elena','Rosa','Catalina','Esteban','Fernando','Eduardo','Javier','Alejandro','Andres','Mateo','Santiago','Sebastian','Emilio','Tomas','Cristina','Daniela','Gabriela','Ximena','Adriana','Beatriz','Pilar','Mercedes','Xavier','Marisol','Guadalupe','Lupita','Inez','Itzel','Yesenia','Joaquin','Ignacio','Rafael','Salvador','Cesar','Arturo','Armando','Hugo','Marco','Alejandra','Felipe','Gerardo','Jaime','Leonardo','Luis','Pablo','Ramon']);
var NAMES_BLACK_C=new Set(['DeShawn','Jamal','Aisha','Latoya','Tyrone','Malik','Imani','Keisha','Tariq','Lakisha','Kenya','Tamika','Andre','Marcus','Demetrius','Jermaine','Reggie','Tyrese','Darius','Trevon','Kareem','Damon','Jalen','Jaylen','Dwayne','DaQuan','Aaliyah','Kiara','Janelle','Jasmine','Tanisha','Maurice','Tyrell','Kwame','Khalil','Terrell','Cedric','Nia','Zuri','Jada','Ebony','Dominique']); var NAMES_BLACK_C=new Set(['DeShawn','Jamal','Aisha','Latoya','Tyrone','Malik','Imani','Keisha','Tariq','Lakisha','Kenya','Tamika','Andre','Marcus','Demetrius','Jermaine','Reggie','Tyrese','Darius','Trevon','Kareem','Damon','Jalen','Jaylen','Dwayne','DaQuan','Aaliyah','Kiara','Janelle','Jasmine','Tanisha','Maurice','Tyrell','Kwame','Khalil','Terrell','Cedric','Nia','Zuri','Jada','Ebony','Dominique']);
var NAMES_MIDDLE_EASTERN_C=new Set(['Layla','Omar','Khalid','Fatima','Yasmin','Hassan','Hussein','Ahmed','Mohamed','Mohammed','Ali','Karim','Yusuf','Yara','Nadia','Zainab','Rania','Samira','Mariam','Salma','Ibrahim','Mahmoud','Saif','Anwar','Bilal','Faisal','Hamza','Imran','Sami','Wael','Zaid','Amira','Iman','Lina','Mona','Noor','Rana','Soha','Zara']); var NAMES_MIDDLE_EASTERN_C=new Set(['Layla','Omar','Khalid','Fatima','Yasmin','Hassan','Hussein','Ahmed','Mohamed','Mohammed','Ali','Karim','Yusuf','Yara','Nadia','Zainab','Rania','Samira','Mariam','Salma','Ibrahim','Mahmoud','Saif','Anwar','Bilal','Faisal','Hamza','Imran','Sami','Wael','Zaid','Amira','Iman','Lina','Mona','Noor','Rana','Soha','Zara']);
// Surname → ethnicity. Surname is more diagnostic than first name
// for hispanic and asian — "Anna Cruz" is hispanic via surname.
var SURNAMES_HISPANIC_C=new Set(['Garcia','Rodriguez','Martinez','Hernandez','Lopez','Gonzalez','Perez','Sanchez','Ramirez','Torres','Flores','Rivera','Gomez','Diaz','Reyes','Cruz','Morales','Ortiz','Gutierrez','Chavez','Ramos','Ruiz','Alvarez','Mendoza','Vasquez','Castillo','Jimenez','Moreno','Romero','Herrera','Medina','Aguilar','Vargas','Castro','Fernandez','Guzman','Munoz','Salazar','Ortega','Delgado','Estrada','Ayala','Pena','Cabrera','Alvarado','Espinoza','Padilla','Cardenas','Cortes','Ibarra','Vega','Soto','Lara','Navarro','Campos','Acosta','Rios','Marquez','Sandoval','Maldonado','Solis','Rojas','Mejia','Beltran','Cervantes','Lozano','Carrillo','Trevino','Robles','Tapia','Lugo']);
var SURNAMES_SOUTH_ASIAN_C=new Set(['Patel','Singh','Kumar','Sharma','Gupta','Shah','Mehta','Desai','Joshi','Reddy','Nair','Iyer','Verma','Agarwal','Kapoor','Chopra','Malhotra','Banerjee','Chatterjee','Mukherjee','Das','Sen','Bose','Roy','Sinha','Trivedi','Pandey','Mishra','Tiwari','Yadav','Chauhan','Rana','Thakur','Pillai','Menon','Krishnan','Rao','Naidu','Pradhan','Acharya','Devi','Kaur']);
var SURNAMES_EAST_ASIAN_C=new Set(['Chen','Wang','Li','Liu','Yang','Huang','Zhao','Wu','Zhou','Xu','Zhu','Sun','Ma','Lin','Lee','Kim','Park','Choi','Jung','Kang','Cho','Yoon','Han','Lim','Oh','Nakamura','Tanaka','Suzuki','Yamamoto','Sato','Watanabe','Takahashi','Kobayashi','Yoshida','Saito','Nguyen','Tran','Le','Pham','Hoang','Phan','Vu','Vo','Dang','Bui','Do','Ngo','Truong','Mai','Cao','Wong','Tang','Tan','Cheng','Lau','Leung','Ng','Cheung','Yip','Hsu','Tsai','Hsieh']);
var SURNAMES_MIDDLE_EASTERN_C=new Set(['Khan','Ahmed','Hussein','Hassan','Ali','Mahmoud','Mohamed','Mohammed','Saleh','Aziz','Karim','Hamad','Najjar','Haddad','Khoury','Mansour','Rahman','Iqbal','Malik','Sheikh','Siddiqui','Qureshi','Saeed']);
function guessEthnicityFromName(first, last){
if(last){
var s=last.replace(/[^A-Za-z]/g,'');
if(s){
var sc=s[0].toUpperCase()+s.slice(1).toLowerCase();
if(SURNAMES_HISPANIC_C.has(sc)) return 'hispanic';
if(SURNAMES_MIDDLE_EASTERN_C.has(sc)) return 'middle_eastern';
if(SURNAMES_SOUTH_ASIAN_C.has(sc)) return 'south_asian';
if(SURNAMES_EAST_ASIAN_C.has(sc)) return 'east_asian';
}
}
if(first){
var clean=first.replace(/[^A-Za-z]/g,'');
if(clean){
var c=clean[0].toUpperCase()+clean.slice(1).toLowerCase();
if(NAMES_MIDDLE_EASTERN_C.has(c)) return 'middle_eastern';
if(NAMES_BLACK_C.has(c)) return 'black';
if(NAMES_HISPANIC_C.has(c)) return 'hispanic';
if(NAMES_SOUTH_ASIAN_C.has(c)) return 'south_asian';
if(NAMES_EAST_ASIAN_C.has(c)) return 'east_asian';
}
}
return 'caucasian';
}
function guessEthnicityFromFirstName(n){ function guessEthnicityFromFirstName(n){
if(!n) return 'caucasian'; if(!n) return 'caucasian';
var clean=n.replace(/[^A-Za-z]/g,''); if(!clean) return 'caucasian'; var clean=n.replace(/[^A-Za-z]/g,''); if(!clean) return 'caucasian';
@@ -317,8 +281,21 @@ function workerRow(name, role, detail, opts){
if(band.band) w.dataset.roleBand = band.band; if(band.band) w.dataset.roleBand = band.band;
var initials = (name||'?').split(' ').map(function(s){return (s[0]||'').toUpperCase()}).join('').substring(0,2); var initials = (name||'?').split(' ').map(function(s){return (s[0]||'').toUpperCase()}).join('').substring(0,2);
var av = el('div','av',initials); var av = el('div','av',initials);
// Headshot insertion removed 2026-04-28. The .av element stays as // Real synthetic headshot via /headshots/<key>; deterministic so
// a monogram-initials avatar. // same worker always gets the same face. Falls back to monogram if
// pool isn't fetched yet.
var faceKey = (opts.face_key) || name || '';
var firstName = (name||'').split(/\s+/)[0]||'';
var gHint = genderFor(firstName);
var eHint = guessEthnicityFromFirstName(firstName);
if(faceKey){
var img=document.createElement('img');
img.alt='';
img.loading='lazy';
img.src = P + '/headshots/' + encodeURIComponent(faceKey) + '?g='+gHint+'&e='+eHint;
img.onerror=function(){ this.remove(); };
av.appendChild(img);
}
w.appendChild(av); w.appendChild(av);
var info = el('div','info'); var info = el('div','info');
var nm = el('div','nm', name||'?'); var nm = el('div','nm', name||'?');
-123
View File
@@ -1,123 +0,0 @@
// Visual filler iconography rendered through ComfyUI. Distinct from
// role_scenes.ts (which renders portraits) — these are object/badge
// style renders that fill dead space on worker cards: cert pills,
// role-prop chips, hazard indicators, empty-state heroes.
//
// Layout on disk:
// data/icons_pool/{category}/{slug}.webp
//
// Cache invalidation:
// ICONS_VERSION mixes into the on-disk filename (slug includes
// version). Bump it after editing a recipe so prior renders are
// ignored on next view.
export type IconCategory = "cert" | "role_prop" | "status" | "hazard" | "empty";
export interface IconRecipe {
slug: string;
category: IconCategory;
// Text label that appears next to / under the icon. The front-end
// already renders this text in cert pills; the icon is supplementary.
display: string;
// Full diffusion prompt. Style guidance baked in. SDXL Turbo at 8
// steps reliably produces clean macro photography, so default to
// photographic prop shots over flat-vector illustrations (the model
// hallucinates noise into flat-vector geometry at low step counts).
prompt: string;
// Negative prompt — what NOT to render. Crucial for icons because
// SDXL likes to add hands/text/people unprompted.
negative?: string;
}
// Default negative prompt baked into every icon render unless the
// recipe overrides. Empirically, these terms are the top SDXL Turbo
// off-style failures.
export const DEFAULT_NEGATIVE =
"people, hands, faces, blurry, low quality, watermark, signature, "
+ "logos, copyright, distorted text, garbled letters, multiple objects";
// TODO J — review and tune the prompts here. Each one is what diffusion
// sees verbatim. The visual decision: photographic prop shots (macro
// photo of an actual badge / placard / sticker) vs flat-icon vector
// style. Default below is photographic — matches the worker headshot
// aesthetic. Flip a recipe to flat-vector by replacing "macro photograph"
// with "flat icon illustration on solid color background, minimal vector".
//
// Visual cues that work well in SDXL Turbo at 8 steps:
// - "macro photograph", "isolated on plain background", "studio lighting"
// - Concrete colors ("orange and black warning diamond") not adjectives
// - Avoid: small text in the prompt (model garbles it), specific brand
// names (creates fake logos), detailed scene composition
const CERT_ICONS: IconRecipe[] = [
{ slug: "osha-10", category: "cert", display: "OSHA-10",
prompt: "macro photograph of a circular yellow safety badge with a black hard hat icon at center, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "osha-30", category: "cert", display: "OSHA-30",
prompt: "macro photograph of a circular orange safety badge with a black hard hat icon at center, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "first-aid-cpr", category: "cert", display: "First Aid/CPR",
prompt: "macro photograph of a small enamel pin badge featuring a bold red cross on a white circular background, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "hazmat", category: "cert", display: "Hazmat",
prompt: "macro photograph of a HAZMAT warning placard, bold orange and black diamond shape with a flame icon, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "forklift", category: "cert", display: "Forklift",
prompt: "macro photograph of a yellow industrial forklift safety badge with a forklift silhouette icon, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "reach-truck", category: "cert", display: "Reach Truck",
prompt: "macro photograph of a navy blue industrial certification badge with a warehouse reach-truck silhouette icon, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "order-picker", category: "cert", display: "Order Picker",
prompt: "macro photograph of a green industrial certification badge with a warehouse order-picker silhouette icon, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "lockout-tagout", category: "cert", display: "Lockout/Tagout",
prompt: "macro photograph of a bright red padlock tag with a danger warning, hanging on a metal industrial valve, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "msds", category: "cert", display: "MSDS",
prompt: "macro photograph of a folded chemical safety data sheet booklet with chemical hazard pictograms visible on cover, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "confined-space", category: "cert", display: "Confined Space",
prompt: "macro photograph of a yellow confined space warning sign featuring a manhole entry icon, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "servsafe", category: "cert", display: "ServSafe",
prompt: "macro photograph of a dark green food safety certification badge featuring a stylized chef hat icon, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "fire-safety", category: "cert", display: "Fire Safety",
prompt: "macro photograph of a red enamel pin badge featuring a flame icon and a fire extinguisher silhouette, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "iso-9001", category: "cert", display: "ISO 9001",
prompt: "macro photograph of a deep blue circular quality-management certification seal with embossed metallic ring, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
];
// Role-band visual chips — small icons that go in the role pill area.
// One per band, optional inline supplement to the existing colored pill.
const ROLE_PROP_ICONS: IconRecipe[] = [
{ slug: "warehouse", category: "role_prop", display: "Warehouse",
prompt: "macro photograph of a yellow hard hat with a high-visibility safety vest folded behind it, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "production", category: "role_prop", display: "Production",
prompt: "macro photograph of a navy blue work shirt and protective safety glasses on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "trades", category: "role_prop", display: "Trades",
prompt: "macro photograph of a leather work glove and a small adjustable wrench on a neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "driver", category: "role_prop", display: "Driver",
prompt: "macro photograph of a navy delivery driver baseball cap and a clipboard manifest on a neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
{ slug: "lead", category: "role_prop", display: "Lead",
prompt: "macro photograph of a tablet showing a bar chart and a high-vis vest folded beside it on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
];
export const ICONS: Record<string, IconRecipe> = Object.fromEntries(
[...CERT_ICONS, ...ROLE_PROP_ICONS].map((r) => [`${r.category}/${r.slug}`, r]),
);
// v2 — 256×256 canvas, intended to be displayed monochrome via CSS
// `filter: grayscale(1)`. Smaller canvas, tighter crops, crisper at
// 14px display size.
export const ICONS_VERSION = "v2";
// Map a free-form cert string from the data ("First Aid/CPR",
// "OSHA-10", "Lockout/Tagout") to the canonical slug used here.
// Returns null if no recipe matches.
export function certToSlug(cert: string): string | null {
const c = (cert || "").trim().toLowerCase().replace(/\s+/g, "-");
if (c === "osha-10") return "osha-10";
if (c === "osha-30") return "osha-30";
if (c.startsWith("first") || c.includes("cpr")) return "first-aid-cpr";
if (c === "hazmat" || c.startsWith("hazwoper")) return "hazmat";
if (c === "forklift" || c.startsWith("pit")) return "forklift";
if (c.startsWith("reach")) return "reach-truck";
if (c.startsWith("order")) return "order-picker";
if (c.startsWith("lockout") || c.includes("tagout")) return "lockout-tagout";
if (c === "msds" || c.startsWith("ghs")) return "msds";
if (c.startsWith("confined")) return "confined-space";
if (c === "servsafe") return "servsafe";
if (c.startsWith("fire")) return "fire-safety";
if (c.startsWith("iso")) return "iso-9001";
return null;
}
+22 -405
View File
@@ -19,17 +19,8 @@ import { SSEServerTransport } from "@modelcontextprotocol/sdk/server/sse.js";
import { z } from "zod"; import { z } from "zod";
import { startTrace, logSpan, logGeneration, scoreTrace, flush as flushTraces } from "./tracing.js"; import { startTrace, logSpan, logGeneration, scoreTrace, flush as flushTraces } from "./tracing.js";
import { buildPermitBrief } from "./entity.js"; import { buildPermitBrief } from "./entity.js";
import { roleBand, SCENES, SCENES_VERSION, FACE_RENDER_DIM, type RoleBand } from "./role_scenes.js";
import { ICONS, ICONS_VERSION, DEFAULT_NEGATIVE, certToSlug, type IconRecipe } from "./icon_recipes.js";
const BASE = process.env.LAKEHOUSE_URL || "http://localhost:3100"; const BASE = process.env.LAKEHOUSE_URL || "http://localhost:3100";
// G5 cutover prep (2026-05-01): when GO_LAKEHOUSE_URL is set, the
// `/_go/*` pass-through routes to the Go gateway. Off-by-default
// (empty = disabled, returns 503 on /_go/*). Reversible via unset.
// Doesn't modify any existing tool — additive parallel path so
// operators can validate Go-side handlers under real Bun-frontend
// shape without touching the production Rust path.
const GO_BASE = process.env.GO_LAKEHOUSE_URL || "";
const PORT = parseInt(process.env.MCP_PORT || "3700"); const PORT = parseInt(process.env.MCP_PORT || "3700");
// ─── Staffer roster — used by the per-staffer hot-swap index (G). ──── // ─── Staffer roster — used by the per-staffer hot-swap index (G). ────
@@ -320,9 +311,9 @@ ${(buckets as any[] || []).map((b: any) => `- ${b.name}: ${b.backend} (${b.reach
- Ollama: :11434 - Ollama: :11434
## Available Models ## Available Models
- qwen3.5:latest: stronger local rung, JSON-clean (default for gen + rerank)
- qwen3: 8.2B, 40K context, thinking+tools (best for reasoning) - qwen3: 8.2B, 40K context, thinking+tools (best for reasoning)
- qwen2.5: 7B, 8K context (legacy — 2026-04 era comparison runs only) - qwen2.5: 7B, 8K context (best for fast SQL generation)
- mistral: 7B, 8K context (general generation)
- nomic-embed-text: 137M (embedding, automatic) - nomic-embed-text: 137M (embedding, automatic)
`; `;
return { contents: [{ uri: uri.href, mimeType: "text/plain", text }] }; return { contents: [{ uri: uri.href, mimeType: "text/plain", text }] };
@@ -718,26 +709,6 @@ async function main() {
return new Response(await r.text(), { status: r.status, headers: { "Content-Type": "application/json" } }); return new Response(await r.text(), { status: r.status, headers: { "Content-Type": "application/json" } });
} }
// G5 cutover slice: pass-through to GO_BASE (Go gateway). Same
// shape as /api/* but points at the Go side. Returns 503 when
// GO_LAKEHOUSE_URL isn't set so callers know the cutover slice
// is off rather than silently routing to Rust. No body
// transformation — caller responsible for sending Go-shaped
// requests (e.g. /v1/embed not /ai/embed).
if (url.pathname.startsWith("/_go/")) {
if (!GO_BASE) {
return err("GO_LAKEHOUSE_URL not set; /_go/* cutover slice is disabled", 503);
}
const path = url.pathname.replace("/_go", "");
const body = req.method !== "GET" ? await req.text() : undefined;
try {
const r = await fetch(`${GO_BASE}${path}`, { method: req.method, headers: { "Content-Type": "application/json" }, body });
return new Response(await r.text(), { status: r.status, headers: { "Content-Type": "application/json" } });
} catch (e) {
return err(`Go gateway unreachable at ${GO_BASE}: ${e}`, 502);
}
}
// Proof — narrative HTML served from mcp-server/proof.html. // Proof — narrative HTML served from mcp-server/proof.html.
// Live tests consumed client-side via /proof.json. // Live tests consumed client-side via /proof.json.
if (url.pathname === "/proof") { if (url.pathname === "/proof") {
@@ -1254,191 +1225,6 @@ async function main() {
// OSHA national, Chicago history, ticker chart, parent link, // OSHA national, Chicago history, ticker chart, parent link,
// federal contracts, debarment, unions, training. Click any // federal contracts, debarment, unions, training. Click any
// contractor name in a permit Entity Brief to land here. // contractor name in a permit Entity Brief to land here.
// ComfyUI-generated portrait — every call is unique by (key,
// gender, race, age, role) tuple. First hit takes ~1.5s on
// the A4000; subsequent hits read from disk. Use this for
// contractor / profile modal where one worker gets the
// spotlight. NB: declared BEFORE the pool route so the prefix
// match doesn't intercept it.
// Single source of truth for the pre-render script. Read
// role_scenes.ts SCENES + SCENES_VERSION so a Python pre-render
// job (scripts/staffing/render_role_pool.py) builds the role-
// aware pool with the exact prompts the server will use on the
// ComfyUI hot-path. No drift.
if (url.pathname === "/headshots/_scenes" && req.method === "GET") {
return Response.json({ version: SCENES_VERSION, scenes: SCENES });
}
// Single source of truth for icon_recipes.ts. Used by the
// pre-render script (scripts/staffing/render_icons.py) and any
// tooling that wants to enumerate available icons.
if (url.pathname === "/icons/_recipes" && req.method === "GET") {
return Response.json({
version: ICONS_VERSION,
default_negative: DEFAULT_NEGATIVE,
recipes: ICONS,
});
}
// Free-text cert resolver: front-end passes the raw cert string
// from the data ("First Aid/CPR", "OSHA-10", "Lockout/Tagout")
// and we resolve to a recipe slug + 302 to the cached/rendered
// icon. Returns 404 (not error) when no recipe matches — the
// front-end can hang an `onerror="this.remove()"` to silently
// drop the img tag for unrecognized certs.
if (url.pathname === "/icons/cert" && req.method === "GET") {
const text = url.searchParams.get("text") || "";
const slug = certToSlug(text);
if (!slug) return new Response(`no recipe for cert: ${text}`, { status: 404 });
return new Response(null, {
status: 302,
headers: { "Location": `/icons/render/cert/${slug}` },
});
}
// Cert / role-prop / status / hazard / empty icons. Lookup is
// category/slug; on cache miss the route renders via ComfyUI.
// Filename layout: data/icons_pool/{category}/{slug}_{version}.webp
// — the version suffix means editing a recipe yields a new file
// rather than overwriting in place, so a misfire is recoverable.
if (url.pathname.startsWith("/icons/render/") && req.method === "GET") {
const rest = url.pathname.slice("/icons/render/".length);
const recipe: IconRecipe | undefined = ICONS[rest];
if (!recipe) return new Response(`unknown icon: ${rest}`, { status: 404 });
const ICONS_DIR = "/home/profit/lakehouse/data/icons_pool";
await Bun.$`mkdir -p ${ICONS_DIR}/${recipe.category}`.quiet();
const cachePath = `${ICONS_DIR}/${recipe.category}/${recipe.slug}_${ICONS_VERSION}.webp`;
const cached = Bun.file(cachePath);
if (await cached.exists()) {
return new Response(cached, {
headers: {
"Content-Type": "image/webp",
"Cache-Control": "public, max-age=86400",
"X-Icon-Source": "cached",
"X-Icon-Recipe": recipe.slug,
},
});
}
// Deterministic seed per recipe — same recipe always renders
// the same icon. Mixing the version means SCENES_VERSION-
// style invalidation works for icons too.
const seedStr = `${recipe.category}|${recipe.slug}|${ICONS_VERSION}`;
let seed = 5381;
for (let i = 0; i < seedStr.length; i++) seed = ((seed << 5) + seed + seedStr.charCodeAt(i)) | 0;
seed = Math.abs(seed) % 2147483647;
try {
const genResp = await fetch("http://localhost:3600/generate", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
prompt: recipe.prompt,
negative_prompt: recipe.negative ?? DEFAULT_NEGATIVE,
// 256×256 — smaller canvas = cleaner icon. SDXL Turbo
// at 8 steps adds visible texture/noise into 512² that
// looks "AI" at small display sizes; tightening to 256
// both renders ~3× faster and produces crisper edges
// when the front-end downsamples to 14px.
width: 256,
height: 256,
steps: 8,
seed,
}),
signal: AbortSignal.timeout(30000),
});
if (!genResp.ok) return new Response(`gen failed: ${genResp.status}`, { status: 502 });
const data: any = await genResp.json();
if (!data.image) return new Response("no image returned", { status: 502 });
const bytes = Uint8Array.from(atob(data.image), (c) => c.charCodeAt(0));
await Bun.write(cachePath, bytes);
return new Response(bytes, {
headers: {
"Content-Type": "image/webp",
"Cache-Control": "public, max-age=86400",
"X-Icon-Source": "fresh",
"X-Icon-Recipe": recipe.slug,
"X-Icon-Gen-Ms": String(data.time_ms || 0),
},
});
} catch (e: any) {
return new Response(`gen error: ${e.message}`, { status: 502 });
}
}
if (url.pathname.startsWith("/headshots/generate/") && req.method === "GET") {
const key = decodeURIComponent(url.pathname.slice("/headshots/generate/".length));
if (!key) return new Response("missing key", { status: 400 });
const g = (url.searchParams.get("g") || "person").toLowerCase();
const r = (url.searchParams.get("e") || "").toLowerCase();
const role = (url.searchParams.get("role") || "warehouse worker").toLowerCase();
const age = parseInt(url.searchParams.get("age") || "32", 10) || 32;
const band = roleBand(role);
// SCENES_VERSION mixes into the cache key so editing
// role_scenes.ts auto-invalidates prior renders — coordinator
// tweaks the warehouse prompt, every warehouse face refreshes
// on next view.
const cacheKey = await crypto.subtle.digest(
"SHA-256",
new TextEncoder().encode(`${key}|${g}|${r}|${role}|${age}|${SCENES_VERSION}`)
).then((b) => Array.from(new Uint8Array(b)).map((x) => x.toString(16).padStart(2, "0")).join("").slice(0, 24));
const GEN_DIR = "/home/profit/lakehouse/data/headshots_gen";
await Bun.$`mkdir -p ${GEN_DIR}`.quiet();
const cachePath = `${GEN_DIR}/${cacheKey}.webp`;
const cached = Bun.file(cachePath);
if (await cached.exists()) {
return new Response(cached, {
headers: {
"Content-Type": "image/webp",
"Cache-Control": "public, max-age=86400, immutable",
"X-Headshot-Source": "comfyui-cached",
},
});
}
const raceText = r === "hispanic" ? "Hispanic"
: r === "black" ? "Black"
: r === "south_asian" ? "South Asian"
: r === "east_asian" ? "East Asian"
: r === "middle_eastern" ? "Middle Eastern"
: "";
const genderText = g === "woman" ? "woman" : g === "man" ? "man" : "person";
const scene = SCENES[band].scene;
// Note: dropped "plain studio background" / "dslr" — those
// collapsed every render to interchangeable studio shots.
// The scene clause now carries clothing + backdrop so a
// forklift operator looks like a forklift operator.
const prompt = `professional headshot portrait of a ${age}-year-old ${raceText} ${genderText} ${role}, ${scene}, neutral confident expression, sharp focus, photorealistic`;
// Worker-derived seed — same input always picks the same
// pixel layout in StyleGAN2 latent space, so the face is
// deterministic per worker BUT distinct from any other
// worker that happens to share the same prompt. Without
// this, every (g, r, age, role) combo collapses to one face.
let seedHash = 0;
for (let i = 0; i < key.length; i++) seedHash = ((seedHash << 5) - seedHash + key.charCodeAt(i)) | 0;
const seed = Math.abs(seedHash) % 2147483647;
try {
const genResp = await fetch("http://localhost:3600/generate", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ prompt, width: FACE_RENDER_DIM, height: FACE_RENDER_DIM, steps: 8, seed }),
signal: AbortSignal.timeout(30000),
});
if (!genResp.ok) return new Response(`gen failed: ${genResp.status}`, { status: 502 });
const data: any = await genResp.json();
if (!data.image) return new Response("no image returned", { status: 502 });
const bytes = Uint8Array.from(atob(data.image), (c) => c.charCodeAt(0));
await Bun.write(cachePath, bytes);
return new Response(bytes, {
headers: {
"Content-Type": "image/webp",
"Cache-Control": "public, max-age=86400, immutable",
"X-Headshot-Source": "comfyui-fresh",
"X-Headshot-Gen-Ms": String(data.time_ms || 0),
},
});
} catch (e: any) {
return new Response(`gen error: ${e.message}`, { status: 502 });
}
}
// Headshot pool — synthetic StyleGAN faces from // Headshot pool — synthetic StyleGAN faces from
// thispersondoesnotexist.com fetched offline by // thispersondoesnotexist.com fetched offline by
// scripts/staffing/fetch_face_pool.py. Deterministic mapping: // scripts/staffing/fetch_face_pool.py. Deterministic mapping:
@@ -1463,47 +1249,19 @@ async function main() {
const raw = await Bun.file(`${HEADSHOT_DIR}/manifest.jsonl`).text(); const raw = await Bun.file(`${HEADSHOT_DIR}/manifest.jsonl`).text();
const lines = raw.trim().split("\n").filter(Boolean); const lines = raw.trim().split("\n").filter(Boolean);
const all = lines.map((l) => JSON.parse(l)); const all = lines.map((l) => JSON.parse(l));
// Build (gender × race) buckets so a request that names
// both narrows to the intersection. Missing intersections
// fall back to gender-only, then race-only, then all.
const byGR: Record<string, any[]> = {};
const byG: Record<string, any[]> = { man: [], woman: [] };
const byR: Record<string, any[]> = {};
// Filter excluded faces (e.g. minors) from every bucket
// and from the all-pool. They never get served.
const adults = all.filter((r: any) => !r.excluded);
for (const r of adults) {
if (r.gender === "man" || r.gender === "woman") byG[r.gender].push(r);
if (r.race) {
byR[r.race] = byR[r.race] || [];
byR[r.race].push(r);
if (r.gender === "man" || r.gender === "woman") {
const k = r.gender + "/" + r.race;
byGR[k] = byGR[k] || [];
byGR[k].push(r);
}
}
}
(globalThis as any)._faces = { (globalThis as any)._faces = {
all: adults, all,
byG, byR, byGR, man: all.filter((r: any) => r.gender === "man"),
untagged: adults.filter((r: any) => !r.gender || (r.gender !== "man" && r.gender !== "woman")), woman: all.filter((r: any) => r.gender === "woman"),
excluded_count: all.length - adults.length, untagged: all.filter((r: any) => !r.gender || (r.gender !== "man" && r.gender !== "woman")),
loaded_at: Date.now(), loaded_at: Date.now(),
}; };
if (key === "__reload") { if (key === "__reload") {
const byRSummary: Record<string, number> = {};
for (const k of Object.keys(byR)) byRSummary[k] = byR[k].length;
const byGRSummary: Record<string, number> = {};
for (const k of Object.keys(byGR)) byGRSummary[k] = byGR[k].length;
return Response.json({ return Response.json({
reloaded: true, reloaded: true,
total: all.length, total: all.length,
excluded: all.length - adults.length, man: (globalThis as any)._faces.man.length,
served_pool: adults.length, woman: (globalThis as any)._faces.woman.length,
by_gender: { man: byG.man.length, woman: byG.woman.length },
by_race: byRSummary,
by_gender_race: byGRSummary,
untagged: (globalThis as any)._faces.untagged.length, untagged: (globalThis as any)._faces.untagged.length,
}); });
} }
@@ -1511,84 +1269,20 @@ async function main() {
return new Response(`face pool not available: ${e.message}. Run scripts/staffing/fetch_face_pool.py first.`, { status: 503 }); return new Response(`face pool not available: ${e.message}. Run scripts/staffing/fetch_face_pool.py first.`, { status: 503 });
} }
} }
const F = (globalThis as any)._faces as { const F = (globalThis as any)._faces as { all: any[]; man: any[]; woman: any[]; untagged: any[] };
all: any[];
byG: Record<string, any[]>;
byR: Record<string, any[]>;
byGR: Record<string, any[]>;
untagged: any[];
};
if (!F || !F.all.length) { if (!F || !F.all.length) {
return new Response("face pool empty", { status: 503 }); return new Response("face pool empty", { status: 503 });
} }
const wantRace = url.searchParams.get("e") || ""; // Pool selection: gender hint > full pool. If no gender match,
// fall back to the full pool so the worker still gets a face.
// NOTE: role-aware pool + ComfyUI sparse redirect were removed
// 2026-04-28 — diffusion output at 8 steps with the existing
// editorial_hero workflow produced burnt-looking faces ("looks
// like someone burnt the pictures"). Until serve_imagegen.py
// is fixed to honor a portrait-friendly negative prompt and
// run with proper steps/cfg, every face comes from the studio
// pool (StyleGAN photos from thispersondoesnotexist.com) and
// gets B&W via CSS. The role pool files at
// data/headshots_role_pool/{v1,v2}/ stay on disk for when
// we can re-enable them.
// Studio pool only. Try gender×race intersection first, then
// fall back to gender-only or race-only if the intersection
// is sparse. Repeat faces are acceptable — better than
// serving the over-contrasty diffusion output.
let pool = F.all; let pool = F.all;
let bucket = "all"; if (wantGender === "man" && F.man.length) pool = F.man;
if (wantGender && wantRace) { else if (wantGender === "woman" && F.woman.length) pool = F.woman;
const gr = F.byGR[wantGender + "/" + wantRace] || [];
if (gr.length > 0) {
// Use the intersection bucket as-is — even sparse buckets
// (south_asian: 3, black: 14) just repeat photos rather
// than route to ComfyUI. Repetition is fine; burnt faces
// are not.
pool = gr;
bucket = `gr:${wantGender}/${wantRace}`;
} else if (F.byG[wantGender]?.length) {
pool = F.byG[wantGender];
bucket = `g:${wantGender}`;
}
} else if (wantGender && F.byG[wantGender]?.length) {
pool = F.byG[wantGender];
bucket = `g:${wantGender}`;
} else if (wantRace && F.byR[wantRace]?.length) {
pool = F.byR[wantRace];
bucket = `r:${wantRace}`;
}
// Hash key → pool index. djb2-ish, fits any string. // Hash key → pool index. djb2-ish, fits any string.
let h = 5381; let h = 5381;
for (let i = 0; i < key.length; i++) h = ((h << 5) + h + key.charCodeAt(i)) | 0; for (let i = 0; i < key.length; i++) h = ((h << 5) + h + key.charCodeAt(i)) | 0;
const idx = Math.abs(h) % pool.length; const idx = Math.abs(h) % pool.length;
const pick = pool[idx]; const pick = pool[idx];
// Prefer pre-resized webp thumb (~10KB) over native JPEG
// (~580KB). 60× smaller — without this, a 40-card grid
// overruns Chrome's parallel-connection budget and ~75% of
// tiles never finish decoding.
//
// Cache-Control: 1h public + must-revalidate, NOT immutable.
// We deliberately let the browser re-check after pool retags
// or face-pool refreshes — `immutable` was pinning stale
// photos for 24h after a server-side update.
const thumbName = pick.file.replace(/\.jpg$/, ".webp");
const thumb = Bun.file(`${HEADSHOT_DIR}/_thumbs/${thumbName}`);
if (await thumb.exists()) {
return new Response(thumb, {
headers: {
"Content-Type": "image/webp",
"Cache-Control": "public, max-age=3600, must-revalidate",
"X-Face-Pool-Idx": String(pick.id),
"X-Face-Pool-Gender": pick.gender || "untagged",
"X-Face-Pool-Bucket": bucket,
"X-Face-Pool-Bucket-Size": String(pool.length),
"X-Face-Pool-Variant": "thumb-384",
},
});
}
const file = Bun.file(`${HEADSHOT_DIR}/${pick.file}`); const file = Bun.file(`${HEADSHOT_DIR}/${pick.file}`);
if (!(await file.exists())) { if (!(await file.exists())) {
return new Response("face missing on disk", { status: 404 }); return new Response("face missing on disk", { status: 404 });
@@ -1596,12 +1290,9 @@ async function main() {
return new Response(file, { return new Response(file, {
headers: { headers: {
"Content-Type": "image/jpeg", "Content-Type": "image/jpeg",
"Cache-Control": "public, max-age=3600, must-revalidate", "Cache-Control": "public, max-age=86400, immutable",
"X-Face-Pool-Idx": String(pick.id), "X-Face-Pool-Idx": String(pick.id),
"X-Face-Pool-Gender": pick.gender || "untagged", "X-Face-Pool-Gender": pick.gender || "untagged",
"X-Face-Pool-Bucket": bucket,
"X-Face-Pool-Bucket-Size": String(pool.length),
"X-Face-Pool-Variant": "native-1024",
}, },
}); });
} }
@@ -2081,88 +1772,15 @@ async function main() {
.reduce((s, c) => s + (c.implied_pay_rate - contractBillRate) * hoursPerWeek * weeksAssumed, 0); .reduce((s, c) => s + (c.implied_pay_rate - contractBillRate) * hoursPerWeek * weeksAssumed, 0);
// Shift inference from permit work_type + description. // Shift inference from permit work_type + description.
// Description keywords trump the hash-based assignment; // Construction defaults to 1st-shift (day). Heavy civil or
// for everything else we deterministically distribute // facility work sometimes runs 2nd or split-shift. 3rd
// permits across shifts via a hash of the permit id so // (overnight) is rare in commercial construction but real
// every shift bucket has real, stable data instead of // for maintenance / emergency calls.
// every contract collapsing to 1st.
const descLower = ((p.work_description || "") + " " + (p.work_type || "")).toLowerCase(); const descLower = ((p.work_description || "") + " " + (p.work_type || "")).toLowerCase();
function hashStr(s: string){ const shifts: string[] = ["1st"]; // default day
let h=5381; if (/night|overnight|24\s*hr|emergency/.test(descLower)) shifts.push("3rd");
for(let i=0;i<s.length;i++) h=((h<<5)+h+s.charCodeAt(i))|0; if (/multi.?shift|round.?the.?clock|double.?shift/.test(descLower)) shifts.push("2nd");
return Math.abs(h); if (/weekend|saturday|sunday/.test(descLower)) shifts.push("4th");
}
const permitKey = String(p.id || (p.street_number+p.street_name) || p.work_description || "").slice(0,80);
const hh = hashStr(permitKey);
const bucket = hh % 100;
// Realistic split: 50% day, 28% evening, 17% overnight,
// 5% weekend. Construction skews heavily day-shift.
let primary: string =
bucket < 50 ? "1st"
: bucket < 78 ? "2nd"
: bucket < 95 ? "3rd"
: "4th";
const shifts: string[] = [primary];
if (/night|overnight|24\s*hr|emergency/.test(descLower) && !shifts.includes("3rd")) shifts.push("3rd");
if (/multi.?shift|round.?the.?clock|double.?shift/.test(descLower) && !shifts.includes("2nd")) shifts.push("2nd");
if (/weekend|saturday|sunday/.test(descLower) && !shifts.includes("4th")) shifts.push("4th");
// Internal calendar: build a 7-day schedule (today ±3
// days) with a row per (date, shift). This is what the
// front-end's shift-mix preview filters against — real
// dates, real workers/bill, real status (past/active/
// scheduled) tied to the current clock. As permits get
// ingested with explicit start/end dates the backend
// can replace this with the stored schedule.
const SHIFT_HOURS: Record<string, [number, number]> = {
"1st": [6, 14], "2nd": [14, 22], "3rd": [22, 30], "4th": [0, 24], // 4th = weekend
};
function shiftStatus(d: Date, shift: string, ref: Date): "past" | "active" | "scheduled" {
const refDay = ref.toISOString().slice(0,10);
const dDay = d.toISOString().slice(0,10);
if (dDay < refDay) return "past";
if (dDay > refDay) return "scheduled";
// Same day — break by hour vs shift window.
const hr = ref.getHours() + ref.getMinutes()/60;
const [s,e] = SHIFT_HOURS[shift] || [0,24];
if (shift === "4th") {
// Weekend shift: active if today IS weekend, else scheduled.
const isWknd = (ref.getDay()===0 || ref.getDay()===6);
return isWknd ? "active" : "scheduled";
}
if (shift === "3rd") {
// 3rd wraps midnight: active 22:00–06:00.
if (hr >= 22 || hr < 6) return "active";
return "scheduled";
}
if (hr < s) return "scheduled";
if (hr >= e) return "past";
return "active";
}
const refNow = new Date();
const schedule: any[] = [];
for (let off = -3; off <= 3; off++) {
const d = new Date(refNow.getTime() + off * 86400e3);
const isWknd = (d.getDay()===0 || d.getDay()===6);
const dateStr = d.toISOString().slice(0,10);
for (const sh of shifts) {
// Weekend permits use 4th shift only; weekday work
// uses its primary shift(s) and skips 4th.
if (isWknd && sh !== "4th") continue;
if (!isWknd && sh === "4th") continue;
// Workers per shift: full count on primary, half on
// secondary so the bill demand differs visibly.
const isPrimary = (sh === primary);
const wForShift = isPrimary ? count : Math.max(1, Math.floor(count/2));
schedule.push({
date: dateStr,
shift: sh,
workers_needed: wForShift,
bill_rate: contractBillRate,
status: shiftStatus(d, sh, refNow),
});
}
}
contracts.push({ contracts.push({
permit: { permit: {
@@ -2217,7 +1835,6 @@ async function main() {
over_bill_pool_margin_at_risk: Math.round(overBillPoolMargin), over_bill_pool_margin_at_risk: Math.round(overBillPoolMargin),
}, },
shifts_needed: shifts, shifts_needed: shifts,
schedule,
}); });
} }
+8 -10
View File
@@ -146,16 +146,15 @@ async function persistOp(op: ObservedOp) {
// ─── LLM Team escalation (code_review mode) ─── // ─── LLM Team escalation (code_review mode) ───
// //
// When recent failures on a single sig_hash cross a threshold the // When recent failures on a single sig_hash cross a threshold the
// local-model analysis is probably insufficient. J's 2026-04-24 // local qwen2.5 analysis is probably insufficient. J's 2026-04-24
// direction: "the observer would trigger to give more context" — // direction: "the observer would trigger to give more context" —
// route failure clusters to LLM Team's specialized code_review mode // route failure clusters to LLM Team's specialized code_review mode
// (via /api/run) so richer structured signal lands in the KB for // (via /api/run) so richer structured signal lands in the KB for
// scrum + auditor + playbook memory to consume next pass. // scrum + auditor + playbook memory to consume next pass.
// //
// Non-destructive: runs in parallel to the existing local diagnose // Non-destructive: runs in parallel to the existing qwen2.5 analysis,
// call (qwen3.5:latest after the 2026-04-30 bump), never replaces // never replaces it. Writes to data/_kb/observer_escalations.jsonl
// it. Writes to data/_kb/observer_escalations.jsonl as a dedicated // as a dedicated audit surface.
// audit surface.
const LLM_TEAM = process.env.LH_LLM_TEAM_URL ?? "http://localhost:5000"; const LLM_TEAM = process.env.LH_LLM_TEAM_URL ?? "http://localhost:5000";
const LLM_TEAM_ESCALATIONS = "/home/profit/lakehouse/data/_kb/observer_escalations.jsonl"; const LLM_TEAM_ESCALATIONS = "/home/profit/lakehouse/data/_kb/observer_escalations.jsonl";
@@ -543,7 +542,7 @@ async function analyzeErrors() {
if (failures.length === 0) return; if (failures.length === 0) return;
// NEW 2026-04-24: escalate recurring sig_hash clusters to LLM Team // NEW 2026-04-24: escalate recurring sig_hash clusters to LLM Team
// code_review mode. Runs in parallel to the local diagnose call // code_review mode. Runs in parallel to the local qwen2.5 analysis
// below — non-blocking, richer downstream signal for scrum/auditor. // below — non-blocking, richer downstream signal for scrum/auditor.
maybeEscalate(failures).catch(() => {}); maybeEscalate(failures).catch(() => {});
@@ -553,14 +552,13 @@ async function analyzeErrors() {
// Ask local model to diagnose. Phase 44 migration (2026-04-27): // Ask local model to diagnose. Phase 44 migration (2026-04-27):
// /v1/chat instead of legacy /ai/generate so /v1/usage tracks the // /v1/chat instead of legacy /ai/generate so /v1/usage tracks the
// call + Langfuse traces it. 2026-04-30 model bump: qwen2.5 → // call + Langfuse traces it. Same upstream model (qwen2.5 local).
// qwen3.5:latest to match the small-model-pipeline local-tier default.
try { try {
const resp = await fetch(`${LAKEHOUSE}/v1/chat`, { const resp = await fetch(`${LAKEHOUSE}/v1/chat`, {
method: "POST", method: "POST",
headers: { "Content-Type": "application/json" }, headers: { "Content-Type": "application/json" },
body: JSON.stringify({ body: JSON.stringify({
model: "qwen3.5:latest", model: "qwen2.5",
provider: "ollama", provider: "ollama",
messages: [{ messages: [{
role: "user", role: "user",
@@ -771,7 +769,7 @@ async function tailOverseerCorrections(): Promise<number> {
try { row = JSON.parse(line); } catch { continue; } try { row = JSON.parse(line); } catch { continue; }
const op: ObservedOp = { const op: ObservedOp = {
timestamp: row.created_at ?? new Date().toISOString(), timestamp: row.created_at ?? new Date().toISOString(),
endpoint: `overseer:${row.model ?? "claude-opus-4-7"}`, endpoint: `overseer:${row.model ?? "gpt-oss:120b"}`,
input_summary: `${row.task_class ?? "?"}: ${row.reason ?? "escalation"}`, input_summary: `${row.task_class ?? "?"}: ${row.reason ?? "escalation"}`,
// Correction itself is neither success nor failure — it's a // Correction itself is neither success nor failure — it's a
// mitigation attempt. We mark success=true so analyzeErrors // mitigation attempt. We mark success=true so analyzeErrors
-92
View File
@@ -1,92 +0,0 @@
// Server-side mirror of search.html's ROLE_BANDS regex table.
// Each band carries a *visual scene* — clothing + immediate backdrop —
// so ComfyUI produces role-coherent headshots instead of interchangeable
// studio portraits. The front-end sends the raw role string in the
// query (?role=Forklift%20Operator); the server resolves it to a band
// and looks up the scene here.
export type RoleBand =
| "warehouse"
| "production"
| "trades"
| "driver"
| "lead";
export interface SceneDef {
band: RoleBand;
// Free-form clause inserted into the diffusion prompt AFTER
// "[age]-year-old [race] [gender] [role], ". Should describe what
// they're wearing and what is immediately behind them. Keep under
// ~25 words — SDXL Turbo loses focus on longer prompts and starts
// hallucinating cartoon hands.
scene: string;
}
const RE_BANDS: { re: RegExp; band: RoleBand }[] = [
{ re: /forklift|warehouse|associate|material\s*handler|loader|loading|packag|shipping|logistics|inventory|sanitation|janit/i, band: "warehouse" },
{ re: /production|assembl|quality/i, band: "production" },
{ re: /welder|weld|electric|maint(enance)?\s*tech|cnc|machine\s*op|hvac|plumb|carpenter|mason|tool\s*&\s*die/i, band: "trades" },
{ re: /driver|truck|haul|cdl/i, band: "driver" },
{ re: /line\s*lead|supervisor|foreman|coordinator|lead\b/i, band: "lead" },
];
export function roleBand(role: string): RoleBand {
const r = (role || "").trim();
if (!r) return "warehouse";
for (const b of RE_BANDS) if (b.re.test(r)) return b.band;
return "warehouse";
}
// TODO J — refine these. Each `scene` string lands directly in the
// diffusion prompt. Tone target: a coordinator glances at the card
// and recognizes the role from the photo before reading the role pill.
//
// Things that work well in SDXL Turbo at 8 steps:
// - One concrete clothing item ("high-visibility yellow vest")
// - One concrete prop ("hard hat hanging from belt", "tablet in hand")
// - One blurred background element ("warehouse pallet aisle behind",
// "factory machinery softly out of focus")
// - Avoid: text/logos (rendered as scribble), specific brands, hands
// holding tools (often distorts), full-body language ("standing",
// "leaning") — model is trained on portrait crops.
//
// Each scene now bakes "monochrome black and white photography" into
// the prompt so the model produces native B&W output rather than us
// applying CSS grayscale post-hoc. SDXL Turbo handles B&W natively
// with strong tonal range — better than desaturating a color render.
export const SCENES: Record<RoleBand, SceneDef> = {
warehouse: {
band: "warehouse",
scene: "wearing a high-visibility safety vest over a t-shirt, hard hat visible, blurred warehouse pallet aisle behind, soft natural light, monochrome black and white photography, fine film grain, documentary portrait style",
},
production: {
band: "production",
scene: "wearing a work shirt with safety glasses on forehead, blurred factory machinery softly out of focus behind, fluorescent overhead lighting, monochrome black and white photography, fine film grain, documentary portrait style",
},
trades: {
band: "trades",
scene: "wearing a heavy-duty work shirt with rolled sleeves, blurred workshop tool wall behind, focused tungsten lighting, monochrome black and white photography, fine film grain, documentary portrait style",
},
driver: {
band: "driver",
scene: "wearing a polo shirt, lanyard with ID badge visible, blurred truck cab or loading dock behind, daylight, monochrome black and white photography, fine film grain, documentary portrait style",
},
lead: {
band: "lead",
scene: "wearing a button-down shirt, tablet held casually at chest level, blurred warehouse floor in soft focus behind, professional lighting, monochrome black and white photography, fine film grain, documentary portrait style",
},
};
// v2 — baked B&W + 1024×1024 render canvas (4× pixels of v1). Larger
// source means downsampling to a 40px avatar packs more detail per
// displayed pixel, hiding the diffusion-y micro-textures that read as
// "AI generated" at small sizes. Server route reads pool from
// data/headshots_role_pool/{SCENES_VERSION}/... so v1 stays available
// for rollback / A-B comparison.
export const SCENES_VERSION = "v2";
// Default render dimensions used by both the on-demand /headshots/
// generate/:key route and the offline render_role_pool.py script. v1
// used 512²; v2 doubles to 1024² (linear 2× = 4× pixels = ~3× GPU
// time on SDXL Turbo).
export const FACE_RENDER_DIM = 1024;
+161 -1308
View File
File diff suppressed because it is too large Load Diff
-178
View File
@@ -1,178 +0,0 @@
// TIF (Tax Increment Financing) district point-in-polygon lookup.
// Given a property's lat/long, returns which Chicago TIF district (if
// any) contains it. TIF districts are public-subsidy zones — a property
// inside one is receiving city tax-increment funding for its build.
// Strong "this project has financial backing" signal for the Project Index.
//
// Data: data/_entity_cache/tif_districts.geojson (Chicago Open Data
// dataset eejr-xtfb, 100 active districts, 3.2MB). Refresh by re-running
// `curl ... eejr-xtfb.geojson > tif_districts.geojson` — districts
// change rarely (only when city council approves new ones or repeals).
//
// Algorithm: classic ray-casting. For each MultiPolygon's outer ring,
// count edge crossings of an east-going horizontal ray from the point.
// Odd crossings = inside. Holes (inner rings) flip the parity. Library-
// free; correct for arbitrary polygons including the irregular Chicago
// shapes which often have many small detours.
import { readFile } from "node:fs/promises";
import { existsSync } from "node:fs";
import { join } from "node:path";
const TIF_GEOJSON = join("/home/profit/lakehouse/data/_entity_cache", "tif_districts.geojson");
type LngLat = [number, number]; // GeoJSON convention: [longitude, latitude]
type Ring = LngLat[];
type Polygon = Ring[]; // outer ring + optional inner rings (holes)
type MultiPolygon = Polygon[];
type TifFeature = {
name: string;
trim_name?: string;
ref?: string;
approval_date?: string;
expiration?: string;
type?: string; // T-1xx etc.
comm_area?: string;
wards?: string;
// Bounding box for quick reject
bbox: { minLon: number; minLat: number; maxLon: number; maxLat: number };
geometry: MultiPolygon;
};
let tifIdx: TifFeature[] | null = null;
function bboxOfMultiPolygon(mp: MultiPolygon): TifFeature["bbox"] {
let minLon = Infinity, minLat = Infinity, maxLon = -Infinity, maxLat = -Infinity;
for (const poly of mp) {
for (const ring of poly) {
for (const [lon, lat] of ring) {
if (lon < minLon) minLon = lon;
if (lat < minLat) minLat = lat;
if (lon > maxLon) maxLon = lon;
if (lat > maxLat) maxLat = lat;
}
}
}
return { minLon, minLat, maxLon, maxLat };
}
async function ensureLoaded(): Promise<TifFeature[]> {
if (tifIdx) return tifIdx;
if (!existsSync(TIF_GEOJSON)) {
tifIdx = [];
return tifIdx;
}
try {
const raw = JSON.parse(await readFile(TIF_GEOJSON, "utf-8"));
const out: TifFeature[] = [];
for (const f of raw.features || []) {
const geom = f.geometry;
if (!geom) continue;
// Normalize Polygon → MultiPolygon for uniform iteration
let mp: MultiPolygon;
if (geom.type === "MultiPolygon") {
mp = geom.coordinates;
} else if (geom.type === "Polygon") {
mp = [geom.coordinates];
} else {
continue;
}
const props = f.properties || {};
out.push({
name: props.name || "Unknown TIF",
trim_name: props.name_trim,
ref: props.ref,
approval_date: props.approval_d,
expiration: props.expiration,
type: props.type,
comm_area: props.comm_area,
wards: props.wards,
bbox: bboxOfMultiPolygon(mp),
geometry: mp,
});
}
tifIdx = out;
return tifIdx;
} catch (e) {
console.warn("[tif] load failed:", (e as Error).message);
tifIdx = [];
return tifIdx;
}
}
// Ray-casting point-in-polygon (single ring). Returns true if (lon, lat)
// is strictly inside the ring. Edge cases (point exactly on a vertex)
// resolve by half-open interval convention; for our use case (Chicago
// boundary precision is ~1m, sites are point queries) this is fine.
function pointInRing(lon: number, lat: number, ring: Ring): boolean {
let inside = false;
const n = ring.length;
for (let i = 0, j = n - 1; i < n; j = i++) {
const [xi, yi] = ring[i];
const [xj, yj] = ring[j];
const intersect =
yi > lat !== yj > lat &&
lon < ((xj - xi) * (lat - yi)) / (yj - yi + 0) + xi;
if (intersect) inside = !inside;
}
return inside;
}
// Polygon = outer ring + holes. Inside outer AND not inside any hole.
function pointInPolygon(lon: number, lat: number, polygon: Polygon): boolean {
if (polygon.length === 0) return false;
if (!pointInRing(lon, lat, polygon[0])) return false;
for (let i = 1; i < polygon.length; i++) {
if (pointInRing(lon, lat, polygon[i])) return false;
}
return true;
}
export type TifMatch = {
name: string;
ref?: string;
approval_date?: string;
expiration?: string;
comm_area?: string;
wards?: string;
};
export async function findTifDistrict(
longitude: number | string | undefined,
latitude: number | string | undefined,
): Promise<TifMatch | null> {
const lon = typeof longitude === "string" ? parseFloat(longitude) : longitude;
const lat = typeof latitude === "string" ? parseFloat(latitude) : latitude;
if (!lon || !lat || isNaN(lon) || isNaN(lat)) return null;
const idx = await ensureLoaded();
if (idx.length === 0) return null;
for (const f of idx) {
// Bbox reject — cheap O(1) skip for the 99% of districts that
// can't possibly contain the point.
const b = f.bbox;
if (lon < b.minLon || lon > b.maxLon || lat < b.minLat || lat > b.maxLat) continue;
// Full point-in-polygon for any polygon in this MultiPolygon
for (const poly of f.geometry) {
if (pointInPolygon(lon, lat, poly)) {
return {
name: f.name,
ref: f.ref,
approval_date: f.approval_date,
expiration: f.expiration,
comm_area: f.comm_area,
wards: f.wards,
};
}
}
}
return null;
}
export async function getTifIndexStats(): Promise<{
total: number;
loaded: boolean;
}> {
const idx = await ensureLoaded();
return { total: idx.length, loaded: idx.length > 0 };
}
@@ -1,6 +1,6 @@
# Phase 6 — Acceptance Gate Report # Phase 6 — Acceptance Gate Report
**Run:** 2026-04-27T15:43:37.943Z **Run:** 2026-04-27T04:54:32.225Z
**Fixture:** `tests/fixtures/distillation/acceptance/` **Fixture:** `tests/fixtures/distillation/acceptance/`
**Temp root:** `/tmp/distillation_phase6_acceptance` **Temp root:** `/tmp/distillation_phase6_acceptance`
**Pipeline run_ids:** `acceptance-run-1-stable` (first) + `acceptance-run-2-stable` (second / hash reproducibility) **Pipeline run_ids:** `acceptance-run-1-stable` (first) + `acceptance-run-2-stable` (second / hash reproducibility)
@@ -40,13 +40,13 @@
| 19 | scratchpad/tree-split case: fixture row materialized into evidence | found | found | ✓ | | 19 | scratchpad/tree-split case: fixture row materialized into evidence | found | found | ✓ |
| 20 | PRD drift case: fixture row materialized | found | found | ✓ | | 20 | PRD drift case: fixture row materialized | found | found | ✓ |
| 21 | hash reproducibility: per-stage output_hash identical across runs | 0 mismatches | all match | ✓ | | 21 | hash reproducibility: per-stage output_hash identical across runs | 0 mismatches | all match | ✓ |
| 22 | hash reproducibility: run_hash identical | 8dfdacee62380ec2... | 8dfdacee62380ec2... | ✓ | | 22 | hash reproducibility: run_hash identical | 3ea12b160ee9099a... | 3ea12b160ee9099a... | ✓ |
## Hash reproducibility detail ## Hash reproducibility detail
run 1 run_hash: `8dfdacee62380ec20b7420d8f8bad3c395822da6eb0b41eeecd356e88fe20bf0` run 1 run_hash: `3ea12b160ee9099a3c52fe6e7fffd3076de7920d2704d24c789260d63cb1a5a2`
run 2 run_hash: `8dfdacee62380ec20b7420d8f8bad3c395822da6eb0b41eeecd356e88fe20bf0` run 2 run_hash: `3ea12b160ee9099a3c52fe6e7fffd3076de7920d2704d24c789260d63cb1a5a2`
**Bit-for-bit identical.** Two runs of the entire pipeline on the same fixture with the same `recorded_at` produce the same outputs. Distillation is deterministic. **Bit-for-bit identical.** Two runs of the entire pipeline on the same fixture with the same `recorded_at` produce the same outputs. Distillation is deterministic.
@@ -1,8 +1,8 @@
# Phase 8 — Full System Audit Report # Phase 8 — Full System Audit Report
**Run:** 2026-04-27T15:43:38.021Z **Run:** 2026-04-27T04:54:32.283Z
**Git commit:** ca7375ea2b178159a0c61bbf62788a2ffa2390e9 **Git commit:** 73f242e3e41c2aa36b35fe9de54742b248915cb5
**Baseline:** 2026-04-27T10:31:44.043Z (d11632a6fae6) **Baseline:** 2026-04-27T04:53:45.796Z (5bdd159966e6)
## Result: **PASS** ✓ ## Result: **PASS** ✓
@@ -26,7 +26,7 @@
| 1 | P0 | recon doc exists | Y | docs/recon/local-distillation-recon.md present | present | ✓ | | 1 | P0 | recon doc exists | Y | docs/recon/local-distillation-recon.md present | present | ✓ |
| 2 | P0 | tier-1 source streams present | — | all 4 tier-1 jsonls on disk | all present | ✓ | | 2 | P0 | tier-1 source streams present | — | all 4 tier-1 jsonls on disk | all present | ✓ |
| 3 | P1 | schema validators pass on fixtures | Y | ≥40 tests, 0 fail | 51 pass, 0 fail | ✓ | | 3 | P1 | schema validators pass on fixtures | Y | ≥40 tests, 0 fail | 51 pass, 0 fail | ✓ |
| 4 | P2 | materializer dry-run completes | Y | >=1 row from each tier-1 source | 1139 read · 82 written · 2 skipped | ✓ | | 4 | P2 | materializer dry-run completes | Y | >=1 row from each tier-1 source | 1073 read · 16 written · 2 skipped | ✓ |
| 5 | P2 | tier-1 sources each materialize ≥1 row | — | 4/4: distilled_facts, scrum_reviews, audit_facts, mode_experiments | 1/4 hit (mode_experiments) | ✓ | | 5 | P2 | tier-1 sources each materialize ≥1 row | — | 4/4: distilled_facts, scrum_reviews, audit_facts, mode_experiments | 1/4 hit (mode_experiments) | ✓ |
| 6 | P3 | on-disk scored-runs distribution non-empty | Y | >=1 accepted | acc=386 part=132 rej=57 hum=480 | ✓ | | 6 | P3 | on-disk scored-runs distribution non-empty | Y | >=1 accepted | acc=386 part=132 rej=57 hum=480 | ✓ |
| 7 | P3 | scored-runs distribution sums positive | — | >0 total | 1055 total | ✓ | | 7 | P3 | scored-runs distribution sums positive | — | >0 total | 1055 total | ✓ |
@@ -38,19 +38,19 @@
| 13 | P5 | latest run (3fa51d66-784c-4c7d-843d-6c48328a608c) has all 5 stage receipts | Y | collect,score,export-rag,export-sft,export-preference | all present | ✓ | | 13 | P5 | latest run (3fa51d66-784c-4c7d-843d-6c48328a608c) has all 5 stage receipts | Y | collect,score,export-rag,export-sft,export-preference | all present | ✓ |
| 14 | P5 | every stage receipt validates against schema | Y | 0 invalid | 0 invalid | ✓ | | 14 | P5 | every stage receipt validates against schema | Y | 0 invalid | 0 invalid | ✓ |
| 15 | P5 | RunSummary validates | Y | valid | valid | ✓ | | 15 | P5 | RunSummary validates | Y | valid | valid | ✓ |
| 16 | P5 | summary.git_commit is 40-char hex | — | match | 68b6697bcb38... (HEAD: ca7375ea2b17...) | ✓ | | 16 | P5 | summary.git_commit is 40-char hex | — | match | 68b6697bcb38... (HEAD: 73f242e3e41c...) | ✓ |
| 17 | P5 | run_hash is sha256 | Y | /^[0-9a-f]{64}$/ | 2336b96c3638982d... | ✓ | | 17 | P5 | run_hash is sha256 | Y | /^[0-9a-f]{64}$/ | 2336b96c3638982d... | ✓ |
| 18 | P6 | acceptance gate passes 22/22 invariants on fixture | Y | PASS — 22/22 | 22/22 (exit=0) | ✓ | | 18 | P6 | acceptance gate passes 22/22 invariants on fixture | Y | PASS — 22/22 | 22/22 (exit=0) | ✓ |
| 19 | P7 | replay validation passes on 3/3 dry-run sample tasks | Y | 3/3 | 3/3 | ✓ | | 19 | P7 | replay validation passes on 3/3 dry-run sample tasks | Y | 3/3 | 3/3 | ✓ |
| 20 | P7 | replay retrieval surfaces ≥1 playbook on each task (when corpus present) | — | ≥1 task with retrieval | 3/3 | ✓ | | 20 | P7 | replay retrieval surfaces ≥1 playbook on each task (when corpus present) | — | ≥1 task with retrieval | 3/3 | ✓ |
| 21 | P7 | escalation loop guard: no path > 2 models | Y | 0 loops | 0 | ✓ | | 21 | P7 | escalation loop guard: no path > 2 models | Y | 0 loops | 0 | ✓ |
| 22 | P7 | replay_runs.jsonl populated by audit run | — | exists with ≥3 rows added | 27 rows total | ✓ | | 22 | P7 | replay_runs.jsonl populated by audit run | — | exists with ≥3 rows added | 21 rows total | ✓ |
## Drift vs prior baseline ## Drift vs prior baseline
| Metric | Baseline | Current | Δ% | Flag | | Metric | Baseline | Current | Δ% | Flag |
|---|---|---|---|---| |---|---|---|---|---|
| p2_evidence_rows | 25 | 82 | 228% | warn | | p2_evidence_rows | 15 | 16 | 7% | ok |
| p2_evidence_skips | 2 | 2 | 0% | ok | | p2_evidence_skips | 2 | 2 | 0% | ok |
| p3_accepted | 386 | 386 | 0% | ok | | p3_accepted | 386 | 386 | 0% | ok |
| p3_partial | 132 | 132 | 0% | ok | | p3_partial | 132 | 132 | 0% | ok |
@@ -61,7 +61,7 @@
| p4_pref_pairs | 83 | 83 | 0% | ok | | p4_pref_pairs | 83 | 83 | 0% | ok |
| p4_total_quarantined | 1325 | 1325 | 0% | ok | | p4_total_quarantined | 1325 | 1325 | 0% | ok |
**1 metric(s) drifted >20% from baseline.** Investigate before treating outputs as stable. All metrics within 20% of baseline — pipeline stable across runs.
## System health status ## System health status
+7 -52
View File
@@ -29,14 +29,8 @@ CACHE_DIR.mkdir(parents=True, exist_ok=True)
WORKFLOW_PATH = "/opt/ComfyUI/workflows/editorial_hero.json" WORKFLOW_PATH = "/opt/ComfyUI/workflows/editorial_hero.json"
def _cache_key(prompt, width, height, steps, seed=None): def _cache_key(prompt, width, height, steps):
# Include seed so callers can vary outputs deterministically without return hashlib.sha256(f"{prompt}|{width}|{height}|{steps}".encode()).hexdigest()[:24]
# the proxy collapsing to a single cached image. None == legacy
# (omitted from the key for backward compatibility).
bits = f"{prompt}|{width}|{height}|{steps}"
if seed is not None:
bits += f"|{seed}"
return hashlib.sha256(bits.encode()).hexdigest()[:24]
def _cache_get(key): def _cache_get(key):
fp = CACHE_DIR / f"{key}.webp" fp = CACHE_DIR / f"{key}.webp"
@@ -46,15 +40,8 @@ def _cache_put(key, img_bytes):
(CACHE_DIR / f"{key}.webp").write_bytes(img_bytes) (CACHE_DIR / f"{key}.webp").write_bytes(img_bytes)
def _comfyui_generate(prompt, width=1024, height=512, steps=8, seed=None, def _comfyui_generate(prompt, width=1024, height=512, steps=8, seed=None):
negative_prompt=None, cfg=None, sampler=None, scheduler=None): """Submit workflow to ComfyUI and wait for result."""
"""Submit workflow to ComfyUI and wait for result.
Optional overrides — when provided, replace the workflow's defaults.
The workflow template at editorial_hero.json was tuned for product
hero shots with a "no humans" negative prompt; portrait callers MUST
pass `negative_prompt` to avoid the model fighting them on faces.
"""
# Load workflow template # Load workflow template
with open(WORKFLOW_PATH) as f: with open(WORKFLOW_PATH) as f:
workflow = json.load(f) workflow = json.load(f)
@@ -64,21 +51,9 @@ def _comfyui_generate(prompt, width=1024, height=512, steps=8, seed=None,
seed = random.randint(0, 2**32) seed = random.randint(0, 2**32)
workflow["3"]["inputs"]["seed"] = seed workflow["3"]["inputs"]["seed"] = seed
workflow["3"]["inputs"]["steps"] = steps workflow["3"]["inputs"]["steps"] = steps
if cfg is not None:
workflow["3"]["inputs"]["cfg"] = cfg
if sampler:
workflow["3"]["inputs"]["sampler_name"] = sampler
if scheduler:
workflow["3"]["inputs"]["scheduler"] = scheduler
workflow["5"]["inputs"]["width"] = width workflow["5"]["inputs"]["width"] = width
workflow["5"]["inputs"]["height"] = height workflow["5"]["inputs"]["height"] = height
workflow["6"]["inputs"]["text"] = prompt workflow["6"]["inputs"]["text"] = prompt
# Node 7 is the negative-prompt CLIPTextEncode. The default is tuned
# for product hero shots and contains "human, person, face, hand,
# fingers, realistic photo of people" — actively sabotaging any
# portrait render. Always overwrite when negative_prompt is given.
if negative_prompt is not None:
workflow["7"]["inputs"]["text"] = negative_prompt
# Submit to ComfyUI # Submit to ComfyUI
payload = json.dumps({"prompt": workflow}).encode() payload = json.dumps({"prompt": workflow}).encode()
@@ -202,20 +177,9 @@ class ImageHandler(BaseHTTPRequestHandler):
height = min(max(int(body.get("height", 720)), 256), 1080) height = min(max(int(body.get("height", 720)), 256), 1080)
steps = min(max(int(body.get("steps", 50)), 1), 80) steps = min(max(int(body.get("steps", 50)), 1), 80)
seed = body.get("seed") seed = body.get("seed")
# Portrait-friendly overrides — None means "use workflow default".
# negative_prompt MUST be passed by portrait callers to avoid
# the workflow's "no humans" baked-in negative.
negative_prompt = body.get("negative_prompt")
cfg = body.get("cfg")
sampler = body.get("sampler")
scheduler = body.get("scheduler")
# Cache check — seed + negative + cfg are part of the key so per- # Cache check
# worker / per-config requests don't collapse to one cached image. key = _cache_key(prompt, width, height, steps)
key = _cache_key(
f"{prompt}||neg={negative_prompt or ''}||cfg={cfg or ''}",
width, height, steps, seed,
)
cached = _cache_get(key) cached = _cache_get(key)
if cached: if cached:
self._json(200, {"image": cached, "format": "webp", "width": width, "height": height, self._json(200, {"image": cached, "format": "webp", "width": width, "height": height,
@@ -228,11 +192,7 @@ class ImageHandler(BaseHTTPRequestHandler):
try: try:
comfy_check = urllib.request.urlopen(f"{COMFYUI_URL}/system_stats", timeout=3) comfy_check = urllib.request.urlopen(f"{COMFYUI_URL}/system_stats", timeout=3)
if comfy_check.status == 200: if comfy_check.status == 200:
img_bytes, seed = _comfyui_generate( img_bytes, seed = _comfyui_generate(prompt, width, height, steps, seed)
prompt, width, height, steps, seed,
negative_prompt=negative_prompt, cfg=cfg,
sampler=sampler, scheduler=scheduler,
)
backend = "comfyui" backend = "comfyui"
except: except:
pass pass
@@ -250,11 +210,6 @@ class ImageHandler(BaseHTTPRequestHandler):
elapsed_ms = int((time.time() - t0) * 1000) elapsed_ms = int((time.time() - t0) * 1000)
img_b64 = base64.b64encode(img_bytes).decode() img_b64 = base64.b64encode(img_bytes).decode()
# Recompute key with the actual seed used (when caller passed
# None, _comfyui_generate picks a random one and we want the
# cache to reflect that so re-requests with the same returned
# seed hit the disk).
key = _cache_key(prompt, width, height, steps, seed)
_cache_put(key, img_bytes) _cache_put(key, img_bytes)
self._json(200, { self._json(200, {
+14 -78
View File
@@ -53,30 +53,20 @@ def fetch_one(idx: int, out_dir: str) -> tuple[int, str, bool, str | None]:
def maybe_tag_gender(records: list[dict], out_dir: str) -> dict[str, int]: def maybe_tag_gender(records: list[dict], out_dir: str) -> dict[str, int]:
"""If deepface is installed, label records that don't already have a """If deepface is installed, label each record with gender. Returns
gender. Returns a count summary; mutates records in place. a count summary; mutates records in place. On import error, returns
None and tags every record as unknown."""
Preservation contract: never overwrites prior `gender` (or any other
tag — race/age/excluded — set by tag_face_pool.py). On deepface
import failure, leaves existing tags alone instead of resetting them
to None. The previous behavior wiped 952 hand-classified rows when
fetch_face_pool was re-run from a Python without deepface installed."""
try: try:
from deepface import DeepFace # type: ignore from deepface import DeepFace # type: ignore
except Exception as e: except Exception as e:
print(f" (deepface unavailable: {e}) — leaving existing tags untouched") print(f" (deepface unavailable: {e}) — pool will mix naturally")
for r in records: for r in records:
r.setdefault("gender", None) r["gender"] = None
already = sum(1 for r in records if r.get("gender") in ("man", "woman")) return {"unknown": len(records)}
return {"preserved_tagged": already, "untagged": len(records) - already}
todo = [r for r in records if r.get("gender") not in ("man", "woman")] print(" tagging gender via deepface (CPU; ~0.5-1s per face)…")
if not todo:
print(" every record already has gender — nothing to tag.")
return {"preserved_tagged": len(records)}
print(f" tagging gender via deepface ({len(todo)} of {len(records)} records, CPU; ~0.5-1s per face)…")
counts: dict[str, int] = {} counts: dict[str, int] = {}
for i, r in enumerate(todo): for i, r in enumerate(records):
full = os.path.join(out_dir, r["file"]) full = os.path.join(out_dir, r["file"])
try: try:
ana = DeepFace.analyze( ana = DeepFace.analyze(
@@ -98,7 +88,7 @@ def maybe_tag_gender(records: list[dict], out_dir: str) -> dict[str, int]:
r["gender_error"] = f"{type(e).__name__}: {e}" r["gender_error"] = f"{type(e).__name__}: {e}"
counts[r["gender"] or "unknown"] = counts.get(r["gender"] or "unknown", 0) + 1 counts[r["gender"] or "unknown"] = counts.get(r["gender"] or "unknown", 0) + 1
if (i + 1) % 25 == 0: if (i + 1) % 25 == 0:
print(f" [{i+1}/{len(todo)}] {counts}") print(f" [{i+1}/{len(records)}] {counts}")
return counts return counts
@@ -111,47 +101,11 @@ def main():
) )
p.add_argument("--concurrency", type=int, default=3, help="parallel fetches (be polite)") p.add_argument("--concurrency", type=int, default=3, help="parallel fetches (be polite)")
p.add_argument("--no-gender", action="store_true", help="skip deepface gender tagging") p.add_argument("--no-gender", action="store_true", help="skip deepface gender tagging")
p.add_argument("--shrink", action="store_true",
help="allow --count to drop manifest entries with id >= count. Default: preserve them.")
args = p.parse_args() args = p.parse_args()
out = os.path.realpath(args.out) out = os.path.realpath(args.out)
os.makedirs(out, exist_ok=True) os.makedirs(out, exist_ok=True)
# Load any existing manifest into a by-id dict so prior tags
# (gender / race / age / excluded) survive the rewrite. Also
# naturally dedupes — if the file accidentally has duplicate
# lines for the same id (this is how we ended up with a 2497-
# row manifest backing a 1000-face pool), the last one wins.
manifest = os.path.join(out, "manifest.jsonl")
existing: dict[int, dict] = {}
if os.path.exists(manifest):
dup_count = 0
with open(manifest) as f:
for line in f:
line = line.strip()
if not line:
continue
try:
row = json.loads(line)
except json.JSONDecodeError:
continue
rid = row.get("id")
if not isinstance(rid, int):
continue
if rid in existing:
dup_count += 1
existing[rid] = row
print(f"Loaded existing manifest: {len(existing)} unique ids ({dup_count} duplicate lines collapsed)")
max_existing = max(existing.keys()) if existing else -1
if max_existing >= args.count and not args.shrink:
print(
f"\nERROR: --count={args.count} would drop {sum(1 for k in existing if k >= args.count)} "
f"manifest entries (max existing id = {max_existing}). Pass --shrink to allow.\n",
file=sys.stderr,
)
sys.exit(2)
print(f"Fetching {args.count} faces → {out}") print(f"Fetching {args.count} faces → {out}")
print(f"Source: {URL} (synthetic StyleGAN — no real people)") print(f"Source: {URL} (synthetic StyleGAN — no real people)")
@@ -161,16 +115,12 @@ def main():
futs = {ex.submit(fetch_one, i, out): i for i in range(args.count)} futs = {ex.submit(fetch_one, i, out): i for i in range(args.count)}
for done, fut in enumerate(as_completed(futs), 1): for done, fut in enumerate(as_completed(futs), 1):
idx, fname, cached, err = fut.result() idx, fname, cached, err = fut.result()
# Start from prior manifest row (preserves gender/race/age/excluded) results[idx] = {
# and overlay only the fields fetch_one is responsible for.
base = dict(existing.get(idx, {}))
base.update({
"id": idx, "id": idx,
"file": fname, "file": fname,
"cached": cached, "cached": cached,
"error": err, "error": err,
}) }
results[idx] = base
if done % 25 == 0 or done == args.count: if done % 25 == 0 or done == args.count:
ok = sum(1 for r in results if r and not r.get("error")) ok = sum(1 for r in results if r and not r.get("error"))
print(f" [{done}/{args.count}] {ok} ok ({time.time()-t0:.1f}s)") print(f" [{done}/{args.count}] {ok} ok ({time.time()-t0:.1f}s)")
@@ -178,9 +128,6 @@ def main():
# Drop slots that errored or are still None (shouldn't happen) # Drop slots that errored or are still None (shouldn't happen)
records = [r for r in results if r and not r.get("error")] records = [r for r in results if r and not r.get("error")]
print(f"\nPool ready: {len(records)} faces, {sum(1 for r in records if r['cached'])} from cache") print(f"\nPool ready: {len(records)} faces, {sum(1 for r in records if r['cached'])} from cache")
preserved_tags = sum(1 for r in records if r.get("gender") in ("man", "woman"))
if preserved_tags:
print(f"Preserved {preserved_tags} prior gender tags (and any race/age/excluded fields).")
if not args.no_gender and records: if not args.no_gender and records:
print("\nGender-tagging pass:") print("\nGender-tagging pass:")
@@ -188,28 +135,17 @@ def main():
print(f" distribution: {summary}") print(f" distribution: {summary}")
else: else:
for r in records: for r in records:
r.setdefault("gender", None) r["gender"] = None
# If --shrink was NOT used and somehow id >= count rows are still in
# `existing` (which can only happen if the early gate was bypassed),
# carry them forward so we don't quietly drop them.
if not args.shrink:
for rid, row in existing.items():
if rid >= args.count and rid not in {r["id"] for r in records}:
records.append(row)
records.sort(key=lambda r: r.get("id", 0))
# Strip transient flags before persisting # Strip transient flags before persisting
for r in records: for r in records:
r.pop("cached", None) r.pop("cached", None)
r.pop("error", None) r.pop("error", None)
# Atomic write — if a re-run is interrupted, manifest stays intact. manifest = os.path.join(out, "manifest.jsonl")
tmp = manifest + ".tmp" with open(manifest, "w") as f:
with open(tmp, "w") as f:
for r in records: for r in records:
f.write(json.dumps(r) + "\n") f.write(json.dumps(r) + "\n")
os.replace(tmp, manifest)
print(f"\nManifest: {manifest} ({len(records)} entries)") print(f"\nManifest: {manifest} ({len(records)} entries)")
# Quick checksum manifest for downstream debugging # Quick checksum manifest for downstream debugging
-230
View File
@@ -1,230 +0,0 @@
#!/usr/bin/env python3
"""
render_role_pool.py — pre-render a role-aware face pool by hitting
serve_imagegen.py (localhost:3600/generate) with prompts pulled from
the bun server's /headshots/_scenes endpoint (single source of truth
for SCENES + SCENES_VERSION).
Layout:
data/headshots_role_pool/
{band}/
{gender}_{race}/
face_00.webp
face_01.webp
...
manifest.jsonl
Each entry in manifest.jsonl:
{"band": "warehouse", "gender": "man", "race": "caucasian",
"file": "warehouse/man_caucasian/face_03.webp",
"seed": 184729338, "scenes_version": "v1"}
Idempotent: a file at the target path is skipped. Re-run with --force
to regenerate. SCENES_VERSION is captured per render so the server's
pool route can refuse stale renders if the version drifts.
"""
from __future__ import annotations
import argparse
import base64
import json
import os
import sys
import time
import urllib.request
import urllib.error
DEFAULT_BANDS = ["warehouse", "production", "trades", "driver", "lead"]
DEFAULT_GENDERS = ["man", "woman"]
DEFAULT_RACES = ["caucasian", "east_asian", "south_asian", "middle_eastern", "black", "hispanic"]
def race_text(r: str) -> str:
return {
"caucasian": "",
"east_asian": "East Asian",
"south_asian": "South Asian",
"middle_eastern": "Middle Eastern",
"black": "Black",
"hispanic": "Hispanic",
}.get(r, "")
def fetch_scenes(mcp_url: str) -> tuple[str, dict]:
"""Pull canonical SCENES from the bun server. Single source of truth."""
req = urllib.request.Request(f"{mcp_url}/headshots/_scenes")
with urllib.request.urlopen(req, timeout=10) as resp:
data = json.loads(resp.read())
return data["version"], data["scenes"]
def render(comfy_url: str, prompt: str, seed: int, steps: int, timeout: int, dim: int) -> bytes | None:
payload = json.dumps({
"prompt": prompt,
"width": dim,
"height": dim,
"steps": steps,
"seed": seed,
}).encode()
req = urllib.request.Request(
f"{comfy_url}/generate",
data=payload,
headers={"Content-Type": "application/json"},
)
try:
with urllib.request.urlopen(req, timeout=timeout) as resp:
data = json.loads(resp.read())
except urllib.error.HTTPError as e:
print(f" HTTP {e.code} from comfy: {e.read()[:200]}", file=sys.stderr)
return None
except Exception as e:
print(f" comfy error: {type(e).__name__}: {e}", file=sys.stderr)
return None
img_b64 = data.get("image")
if not img_b64:
print(f" comfy response missing 'image' field: {list(data.keys())}", file=sys.stderr)
return None
return base64.b64decode(img_b64)
def main():
p = argparse.ArgumentParser()
p.add_argument("--out", default=os.path.join(os.path.dirname(__file__), "..", "..", "data", "headshots_role_pool"))
p.add_argument("--per-bucket", type=int, default=10, help="how many faces per (band × gender × race)")
p.add_argument("--mcp", default="http://localhost:3700")
p.add_argument("--comfy", default="http://localhost:3600")
p.add_argument("--steps", type=int, default=8)
p.add_argument("--bands", nargs="*", default=DEFAULT_BANDS)
p.add_argument("--genders", nargs="*", default=DEFAULT_GENDERS)
p.add_argument("--races", nargs="*", default=DEFAULT_RACES)
p.add_argument("--force", action="store_true", help="regenerate existing files")
p.add_argument("--age", type=int, default=32)
p.add_argument("--timeout", type=int, default=120, help="per-render timeout (1024² takes ~5s on A4000)")
p.add_argument("--dim", type=int, default=1024, help="square render dimension (v2 default 1024, v1 was 512)")
args = p.parse_args()
out_root = os.path.realpath(args.out)
os.makedirs(out_root, exist_ok=True)
print(f"Fetching canonical SCENES from {args.mcp}/headshots/_scenes…")
try:
version, scenes = fetch_scenes(args.mcp)
except Exception as e:
print(f"FATAL: could not fetch scenes ({e}). Is the mcp-server up?", file=sys.stderr)
sys.exit(1)
print(f" SCENES_VERSION={version}, {len(scenes)} bands available: {list(scenes.keys())}")
# v2+ files live at {out}/{version}/{band}/{g}_{r}/face_NN.webp.
# v1 lived at {out}/{band}/... — keep that layout intact for
# rollback; the server route reads both and prefers current.
out = out_root if version == "v1" else os.path.join(out_root, version)
os.makedirs(out, exist_ok=True)
print(f" writing to: {out}")
print(f" render dim: {args.dim}×{args.dim}")
# Reject any --bands not in the server's SCENES
unknown = [b for b in args.bands if b not in scenes]
if unknown:
print(f"FATAL: unknown bands {unknown}. Server has: {list(scenes.keys())}", file=sys.stderr)
sys.exit(1)
manifest_rows = []
todo = [
(band, g, r, n)
for band in args.bands
for g in args.genders
for r in args.races
for n in range(args.per_bucket)
]
print(f"\nPlanning: {len(todo)} renders ({len(args.bands)} bands × {len(args.genders)} genders × {len(args.races)} races × {args.per_bucket} faces).")
print(f"Estimated GPU time at 1.5s/render = {len(todo) * 1.5 / 60:.1f} min.\n")
t0 = time.time()
rendered = 0
skipped = 0
failed = 0
for i, (band, g, r, n) in enumerate(todo):
bucket_dir = os.path.join(out, band, f"{g}_{r}")
os.makedirs(bucket_dir, exist_ok=True)
fname = f"face_{n:02d}.webp"
full = os.path.join(bucket_dir, fname)
rel = os.path.relpath(full, out)
if os.path.exists(full) and os.path.getsize(full) > 1024 and not args.force:
skipped += 1
manifest_rows.append({
"band": band, "gender": g, "race": r, "file": rel,
"seed": None, "scenes_version": version, "cached": True,
})
continue
scene_def = scenes[band]
scene_clause = scene_def["scene"]
race_clause = race_text(r)
gender_clause = g # "man" / "woman"
# Match the bun server's prompt builder exactly. If you tweak
# one, tweak the other (or factor a /prompt-builder endpoint).
# The {role} slot is intentionally a band-typical title here
# — the pre-rendered face is shared across roles in the same
# band, so we use the band's archetypal role. Specific roles
# still hit the on-demand /headshots/generate/:key path with
# their actual title.
archetype_role = {
"warehouse": "warehouse worker",
"production": "production worker",
"trades": "skilled tradesperson",
"driver": "delivery driver",
"lead": "shift supervisor",
}.get(band, "warehouse worker")
prompt = (
f"professional headshot portrait of a {args.age}-year-old "
f"{race_clause} {gender_clause} {archetype_role}, {scene_clause}, "
f"neutral confident expression, sharp focus, photorealistic"
)
# Deterministic seed per slot — same (band, g, r, n) always
# gets the same face. Mixing scenes_version means a SCENES
# tweak shifts every face slightly; that's the right behavior
# (it's how cache invalidation propagates to the pool too).
seed_str = f"{band}|{g}|{r}|{n}|{version}"
seed_h = 5381
for ch in seed_str:
seed_h = ((seed_h << 5) + seed_h + ord(ch)) & 0x7fffffff
seed = seed_h
bytes_ = render(args.comfy, prompt, seed, args.steps, args.timeout, args.dim)
if bytes_ is None:
failed += 1
continue
with open(full, "wb") as f:
f.write(bytes_)
rendered += 1
manifest_rows.append({
"band": band, "gender": g, "race": r, "file": rel,
"seed": seed, "scenes_version": version, "cached": False,
})
if (i + 1) % 10 == 0 or (i + 1) == len(todo):
elapsed = time.time() - t0
done = i + 1
rate = done / elapsed if elapsed > 0 else 0
eta = (len(todo) - done) / rate if rate > 0 else 0
print(f" [{done}/{len(todo)}] rendered={rendered} skipped={skipped} failed={failed} "
f"rate={rate:.2f}/s eta={eta:.0f}s")
# Atomic manifest write
manifest_path = os.path.join(out, "manifest.jsonl")
tmp = manifest_path + ".tmp"
with open(tmp, "w") as f:
for row in manifest_rows:
f.write(json.dumps(row) + "\n")
os.replace(tmp, manifest_path)
print(f"\nDone. {rendered} new, {skipped} cached, {failed} failed in {time.time()-t0:.1f}s")
print(f"Manifest: {manifest_path} ({len(manifest_rows)} entries)")
print(f"\nNext: poke {args.mcp}/headshots/__reload to pick up the new pool.")
if __name__ == "__main__":
main()
-169
View File
@@ -1,169 +0,0 @@
#!/usr/bin/env python3
"""
tag_face_pool.py — run deepface gender + race classification over the
synthetic face pool produced by fetch_face_pool.py and rewrite
manifest.jsonl with `gender` (man / woman) and `race` (asian / black /
hispanic / indian / middle_eastern / white) tags.
Run with the venv that has deepface installed:
/home/profit/.local/share/deepface-venv/bin/python \
scripts/staffing/tag_face_pool.py
Idempotent: rows that already have BOTH gender and race tagged are
skipped. Pass --force to re-tag everything.
Mapping deepface buckets → /headshots/ ?e= values:
asian → split by manual region (deepface doesn't differentiate
East / South Asian; we lump as 'east_asian' since the
StyleGAN training set leans East Asian)
indian → south_asian
middle eastern → middle_eastern
black → black
hispanic → hispanic
white → caucasian
"""
from __future__ import annotations
import argparse
import json
import os
import sys
import time
DEEPFACE_RACE_TO_HINT = {
"asian": "east_asian",
"indian": "south_asian",
"middle eastern": "middle_eastern",
"black": "black",
"latino hispanic": "hispanic",
"hispanic": "hispanic",
"white": "caucasian",
}
def main():
p = argparse.ArgumentParser()
p.add_argument(
"--out",
default=os.path.join(os.path.dirname(__file__), "..", "..", "data", "headshots"),
)
p.add_argument("--force", action="store_true", help="re-tag rows that already have gender+race")
p.add_argument("--limit", type=int, default=0, help="cap how many faces to process this run (0 = all)")
p.add_argument("--min-age", type=int, default=22, help="exclude faces estimated below this age (kids/teens). Staffing context = legal-age workers only.")
args = p.parse_args()
out = os.path.realpath(args.out)
manifest_path = os.path.join(out, "manifest.jsonl")
if not os.path.exists(manifest_path):
print(f"manifest not found: {manifest_path}", file=sys.stderr)
sys.exit(1)
print(f"loading deepface (cold start ~10-15s for first model build)…")
from deepface import DeepFace # type: ignore
rows = []
with open(manifest_path) as f:
for line in f:
line = line.strip()
if not line:
continue
rows.append(json.loads(line))
print(f"manifest: {len(rows)} rows")
todo = [
r for r in rows
if args.force or r.get("gender") is None or r.get("race") is None or r.get("age") is None
]
if args.limit > 0:
todo = todo[: args.limit]
print(f"to tag: {len(todo)} faces")
if not todo:
print("nothing to do.")
return
counts_g = {}
counts_r = {}
failed = 0
t0 = time.time()
for i, r in enumerate(todo):
full = os.path.join(out, r["file"])
try:
ana = DeepFace.analyze(
img_path=full,
actions=["gender", "race", "age"],
enforce_detection=False,
silent=True,
)
if isinstance(ana, list):
ana = ana[0] if ana else {}
g_raw = (ana.get("dominant_gender") or "").lower().strip()
r["gender"] = (
"man" if g_raw.startswith("man") else
"woman" if g_raw.startswith("woman") else
None
)
r_raw = (ana.get("dominant_race") or "").lower().strip()
r["race"] = DEEPFACE_RACE_TO_HINT.get(r_raw, None)
if r["race"] is None and r_raw:
r["race_raw"] = r_raw
# Age estimation — exclude minors / teens. Staffing context
# uses adult workers only. Threshold is 22 by default
# (legal + a buffer because age estimation is noisy).
try:
age = int(round(float(ana.get("age") or 0)))
except Exception:
age = 0
r["age"] = age
if age and age < args.min_age:
r["excluded"] = "minor"
else:
r.pop("excluded", None)
counts_g[r["gender"] or "unknown"] = counts_g.get(r["gender"] or "unknown", 0) + 1
counts_r[r["race"] or r_raw or "unknown"] = counts_r.get(r["race"] or r_raw or "unknown", 0) + 1
except Exception as e:
r["tag_error"] = f"{type(e).__name__}: {e}"
failed += 1
if (i + 1) % 25 == 0 or (i + 1) == len(todo):
elapsed = time.time() - t0
rate = (i + 1) / elapsed if elapsed > 0 else 0
eta = (len(todo) - i - 1) / rate if rate > 0 else 0
print(f" [{i+1}/{len(todo)}] rate={rate:.1f}/s eta={eta:.0f}s failed={failed}")
print(f" gender: {counts_g}")
print(f" race : {counts_r}")
# Write updated manifest atomically
tmp = manifest_path + ".tmp"
with open(tmp, "w") as f:
for r in rows:
f.write(json.dumps(r) + "\n")
os.replace(tmp, manifest_path)
final_g = {}
final_r = {}
excluded = 0
age_hist = {"<18": 0, "18-22": 0, "22-30": 0, "30-40": 0, "40-50": 0, "50-60": 0, "60+": 0, "unknown": 0}
for r in rows:
if r.get("excluded"):
excluded += 1
continue
final_g[r.get("gender") or "untagged"] = final_g.get(r.get("gender") or "untagged", 0) + 1
final_r[r.get("race") or "untagged"] = final_r.get(r.get("race") or "untagged", 0) + 1
a = r.get("age") or 0
if a == 0: age_hist["unknown"] += 1
elif a < 18: age_hist["<18"] += 1
elif a < 22: age_hist["18-22"] += 1
elif a < 30: age_hist["22-30"] += 1
elif a < 40: age_hist["30-40"] += 1
elif a < 50: age_hist["40-50"] += 1
elif a < 60: age_hist["50-60"] += 1
else: age_hist["60+"] += 1
print(f"\nDone. {len(rows)} rows, {excluded} excluded as <{args.min_age}, {failed} tag errors, {time.time()-t0:.1f}s")
print(f" final gender: {final_g}")
print(f" final race : {final_r}")
print(f" age dist : {age_hist}")
print(f"\nNext: poke /headshots/__reload to refresh the in-memory pool.")
if __name__ == "__main__":
main()
+2 -13
View File
@@ -1143,15 +1143,9 @@ Format each as a code-fenced block with the byte offset within the shard:
EXACT LINE OF SOURCE — DO NOT PARAPHRASE, DO NOT TRUNCATE EXACT LINE OF SOURCE — DO NOT PARAPHRASE, DO NOT TRUNCATE
\`\`\` \`\`\`
Pick the most reviewer-relevant lines: route definitions (e.g. \`@app.route(...)\`), function signatures, security-sensitive calls (auth/SQL/exec/template/secrets), hardcoded credentials/defaults, exception handlers, sensitive imports. The reviewer will REFUSE to act on any claim not backed by a verbatim anchor — so anchors are how you prove findings are real.`; Pick the most reviewer-relevant lines: route definitions (e.g. \`@app.route(...)\`), function signatures, security-sensitive calls (auth/SQL/exec/template/secrets), hardcoded credentials/defaults, exception handlers, sensitive imports. The reviewer will REFUSE to act on any claim not backed by a verbatim anchor — so anchors are how you prove findings are real.`;
// 2026-04-28: gpt-oss:120b → gemini-3-flash-preview via Ollama
// Pro. Tree-split MAP fires once per shard (potentially 5-20×
// per file), so latency dominates total scrum time. Gemini 3
// flash returns shard digests substantially faster than the old
// 120B free model while staying strong enough for byte-anchored
// extraction.
const r = await chat({ const r = await chat({
provider: "ollama_cloud", provider: "ollama_cloud",
model: "gemini-3-flash-preview", model: "gpt-oss:120b",
prompt, prompt,
max_tokens: 900, max_tokens: 900,
}); });
@@ -1201,14 +1195,9 @@ COPY EVERY anchor block from the piece notes IN ORDER, character-perfect. DO NOT
Output the anchor blocks under their original \`\`\`@offset...\`\`\` fences, each on its own with a blank line between. The reviewer rejects findings that don't quote a string from this anchors block, so completeness here directly determines review quality.`; Output the anchor blocks under their original \`\`\`@offset...\`\`\` fences, each on its own with a blank line between. The reviewer rejects findings that don't quote a string from this anchors block, so completeness here directly determines review quality.`;
// 2026-04-28: gpt-oss:120b → gemini-3-flash-preview via Ollama
// Pro. The reducer runs once per file (vs once per shard for MAP)
// but on a much larger context (all shard digests stacked), so
// throughput per token still matters. Same model as MAP for
// consistency in tree-split outputs.
const reduced = await chat({ const reduced = await chat({
provider: "ollama_cloud", provider: "ollama_cloud",
model: "gemini-3-flash-preview", model: "gpt-oss:120b",
prompt: reducePrompt, prompt: reducePrompt,
max_tokens: 2400, max_tokens: 2400,
}); });