profit/lakehouse
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Synthetic face pool — 1000 StyleGAN headshots, ComfyUI hot-swap, 60x smaller thumbs

Browse Source 
Worker cards now ship a real photo per person instead of monogram tiles:

  - fetch_face_pool.py pulls 1000 faces from thispersondoesnotexist.com
  - tag_face_pool.py runs deepface for gender/race/age, excludes <22yo
  - manifest.jsonl: 952 servable, gender/race buckets populated
  - /headshots/_thumbs/ pre-resized to 384px webp (587KB -> 11KB,
    60x smaller; without this Chrome's parallel-connection budget
    drops ~75% of tiles in a 40-card grid)
  - /headshots/:key gender x race x age intersection bucketing with
    gender-only fallback when intersection is sparse
  - /headshots/generate/:key ComfyUI on-demand for the contractor
    profile spotlight (cold ~1.5s, cached ~1ms; worker-derived
    djb2 seed makes faces deterministic-per-worker but unique
    across workers sharing the same prompt)
  - serve_imagegen.py _cache_key() now includes seed (was caching
    by prompt only -> 3 different worker seeds collapsed to 1
    cached image; verified fix produces 3 distinct md5s)
  - confidence-default name resolution: Xavier->man+hispanic,
    Aisha->woman+black, etc. Every worker resolves to a bucket.

End-to-end: playwright run on /?q=forklift+operators+IL -> 21/21
cards loaded, 0 broken, all 384px webp.

Cache + binary pool gitignored; manifest tracked.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-28 06:01:04 -05:00
co-authored by Claude Opus 4.7
parent 10ed3bc630
commit a3b65f314e
8 changed files with 1575 additions and 216 deletions
Download Patch File Download Diff File Expand all files Collapse all files
mcp-server/index.ts
+141 -11
View File
@@ -1225,6 +1225,77 @@ async function main() {
// OSHA national, Chicago history, ticker chart, parent link,
// federal contracts, debarment, unions, training. Click any
// contractor name in a permit Entity Brief to land here.
// ComfyUI-generated portrait — every call is unique by (key,
// gender, race, age, role) tuple. First hit takes ~1.5s on
// the A4000; subsequent hits read from disk. Use this for
// contractor / profile modal where one worker gets the
// spotlight. NB: declared BEFORE the pool route so the prefix
// match doesn't intercept it.
if (url.pathname.startsWith("/headshots/generate/") && req.method === "GET") {
const key = decodeURIComponent(url.pathname.slice("/headshots/generate/".length));
if (!key) return new Response("missing key", { status: 400 });
const g = (url.searchParams.get("g") || "person").toLowerCase();
const r = (url.searchParams.get("e") || "").toLowerCase();
const role = (url.searchParams.get("role") || "warehouse worker").toLowerCase();
const age = parseInt(url.searchParams.get("age") || "32", 10) || 32;
const cacheKey = await crypto.subtle.digest(
"SHA-256",
new TextEncoder().encode(`${key}|${g}|${r}|${role}|${age}`)
).then((b) => Array.from(new Uint8Array(b)).map((x) => x.toString(16).padStart(2, "0")).join("").slice(0, 24));
const GEN_DIR = "/home/profit/lakehouse/data/headshots_gen";
await Bun.$`mkdir -p ${GEN_DIR}`.quiet();
const cachePath = `${GEN_DIR}/${cacheKey}.webp`;
const cached = Bun.file(cachePath);
if (await cached.exists()) {
return new Response(cached, {
headers: {
"Content-Type": "image/webp",
"Cache-Control": "public, max-age=86400, immutable",
"X-Headshot-Source": "comfyui-cached",
},
});
}
const raceText = r === "hispanic" ? "Hispanic"
: r === "black" ? "Black"
: r === "south_asian" ? "South Asian"
: r === "east_asian" ? "East Asian"
: r === "middle_eastern" ? "Middle Eastern"
: "";
const genderText = g === "woman" ? "woman" : g === "man" ? "man" : "person";
const prompt = `professional corporate headshot portrait of a ${age}-year-old ${raceText} ${genderText}, ${role}, neutral expression, plain studio background, soft natural lighting, sharp focus, photorealistic, dslr`;
// Worker-derived seed — same input always picks the same
// pixel layout in StyleGAN2 latent space, so the face is
// deterministic per worker BUT distinct from any other
// worker that happens to share the same prompt. Without
// this, every (g, r, age, role) combo collapses to one face.
let seedHash = 0;
for (let i = 0; i < key.length; i++) seedHash = ((seedHash << 5) - seedHash + key.charCodeAt(i)) | 0;
const seed = Math.abs(seedHash) % 2147483647;
try {
const genResp = await fetch("http://localhost:3600/generate", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ prompt, width: 512, height: 512, steps: 8, seed }),
signal: AbortSignal.timeout(30000),
});
if (!genResp.ok) return new Response(`gen failed: ${genResp.status}`, { status: 502 });
const data: any = await genResp.json();
if (!data.image) return new Response("no image returned", { status: 502 });
const bytes = Uint8Array.from(atob(data.image), (c) => c.charCodeAt(0));
await Bun.write(cachePath, bytes);
return new Response(bytes, {
headers: {
"Content-Type": "image/webp",
"Cache-Control": "public, max-age=86400, immutable",
"X-Headshot-Source": "comfyui-fresh",
"X-Headshot-Gen-Ms": String(data.time_ms || 0),
},
});
} catch (e: any) {
return new Response(`gen error: ${e.message}`, { status: 502 });
}
}
// Headshot pool — synthetic StyleGAN faces from
// thispersondoesnotexist.com fetched offline by
// scripts/staffing/fetch_face_pool.py. Deterministic mapping:
@@ -1249,19 +1320,47 @@ async function main() {
const raw = await Bun.file(`${HEADSHOT_DIR}/manifest.jsonl`).text();
const lines = raw.trim().split("\n").filter(Boolean);
const all = lines.map((l) => JSON.parse(l));
// Build (gender × race) buckets so a request that names
// both narrows to the intersection. Missing intersections
// fall back to gender-only, then race-only, then all.
const byGR: Record<string, any[]> = {};
const byG: Record<string, any[]> = { man: [], woman: [] };
const byR: Record<string, any[]> = {};
// Filter excluded faces (e.g. minors) from every bucket
// and from the all-pool. They never get served.
const adults = all.filter((r: any) => !r.excluded);
for (const r of adults) {
if (r.gender === "man" || r.gender === "woman") byG[r.gender].push(r);
if (r.race) {
byR[r.race] = byR[r.race] || [];
byR[r.race].push(r);
if (r.gender === "man" || r.gender === "woman") {
const k = r.gender + "/" + r.race;
byGR[k] = byGR[k] || [];
byGR[k].push(r);
}
}
}
(globalThis as any)._faces = {
all,
man: all.filter((r: any) => r.gender === "man"),
woman: all.filter((r: any) => r.gender === "woman"),
untagged: all.filter((r: any) => !r.gender || (r.gender !== "man" && r.gender !== "woman")),
all: adults,
byG, byR, byGR,
untagged: adults.filter((r: any) => !r.gender || (r.gender !== "man" && r.gender !== "woman")),
excluded_count: all.length - adults.length,
loaded_at: Date.now(),
};
if (key === "__reload") {
const byRSummary: Record<string, number> = {};
for (const k of Object.keys(byR)) byRSummary[k] = byR[k].length;
const byGRSummary: Record<string, number> = {};
for (const k of Object.keys(byGR)) byGRSummary[k] = byGR[k].length;
return Response.json({
reloaded: true,
total: all.length,
man: (globalThis as any)._faces.man.length,
woman: (globalThis as any)._faces.woman.length,
excluded: all.length - adults.length,
served_pool: adults.length,
by_gender: { man: byG.man.length, woman: byG.woman.length },
by_race: byRSummary,
by_gender_race: byGRSummary,
untagged: (globalThis as any)._faces.untagged.length,
});
}
@@ -1269,20 +1368,50 @@ async function main() {
return new Response(`face pool not available: ${e.message}. Run scripts/staffing/fetch_face_pool.py first.`, { status: 503 });
}
}
const F = (globalThis as any)._faces as { all: any[]; man: any[]; woman: any[]; untagged: any[] };
const F = (globalThis as any)._faces as {
all: any[];
byG: Record<string, any[]>;
byR: Record<string, any[]>;
byGR: Record<string, any[]>;
untagged: any[];
};
if (!F || !F.all.length) {
return new Response("face pool empty", { status: 503 });
}
// Pool selection: gender hint > full pool. If no gender match,
// fall back to the full pool so the worker still gets a face.
// Pool selection: try gender×race intersection first, then
// gender-only, then race-only, then full pool. Always returns
// a face so the worker card never falls back to the monogram.
const wantRace = url.searchParams.get("e") || "";
let pool = F.all;
if (wantGender === "man" && F.man.length) pool = F.man;
else if (wantGender === "woman" && F.woman.length) pool = F.woman;
if (wantGender && wantRace && F.byGR[wantGender + "/" + wantRace]?.length) {
pool = F.byGR[wantGender + "/" + wantRace];
} else if (wantGender && F.byG[wantGender]?.length) {
pool = F.byG[wantGender];
} else if (wantRace && F.byR[wantRace]?.length) {
pool = F.byR[wantRace];
}
// Hash key → pool index. djb2-ish, fits any string.
let h = 5381;
for (let i = 0; i < key.length; i++) h = ((h << 5) + h + key.charCodeAt(i)) | 0;
const idx = Math.abs(h) % pool.length;
const pick = pool[idx];
// Prefer pre-resized webp thumb (~10KB) over native JPEG
// (~580KB). 60× smaller — without this, a 40-card grid
// overruns Chrome's parallel-connection budget and ~75% of
// tiles never finish decoding.
const thumbName = pick.file.replace(/\.jpg$/, ".webp");
const thumb = Bun.file(`${HEADSHOT_DIR}/_thumbs/${thumbName}`);
if (await thumb.exists()) {
return new Response(thumb, {
headers: {
"Content-Type": "image/webp",
"Cache-Control": "public, max-age=86400, immutable",
"X-Face-Pool-Idx": String(pick.id),
"X-Face-Pool-Gender": pick.gender || "untagged",
"X-Face-Pool-Variant": "thumb-384",
},
});
}
const file = Bun.file(`${HEADSHOT_DIR}/${pick.file}`);
if (!(await file.exists())) {
return new Response("face missing on disk", { status: 404 });
@@ -1293,6 +1422,7 @@ async function main() {
"Cache-Control": "public, max-age=86400, immutable",
"X-Face-Pool-Idx": String(pick.id),
"X-Face-Pool-Gender": pick.gender || "untagged",
"X-Face-Pool-Variant": "native-1024",
},
});
}