profit/lakehouse
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

demo: profiler index — directory of every Chicago contractor

Browse Source 
J asked for "a profiler index that shows a history of everyone." This
is a /profiler directory page (also reachable via /contractors) that
ranks every contractor who's filed a Chicago permit, by total permit
value. Rows are clickable into the full /contractor profile.

Defaults: since 2025-06-01, min permit cost $250K, top 200 contractors
by total_cost. Server pulls two Socrata GROUP BY queries (one keyed on
contact_1_name, one on contact_2_name), merges them so contractors
listed in either applicant or contractor slot appear once with combined
counts/cost. ~300ms cold.

UI: live search box, since-date selector, min-cost selector, sortable
columns (name / permits / total_cost / last_filed). Live numbers as of
this write: 200 contractors, 1,702 permits, $14.22B aggregate. Filter
"Target" returns TARGET CORPORATION + CORPORATION TARGET (name variants
from Socrata).

Also fixes J's other complaint — "no new contracts, Target is gone":

  /intelligence/permit_contracts was hard-capped at $limit=6 + only
  the most recent 6 over $250K, so any day with 6 fresh permits would
  push older contractors (Target) off the panel entirely. Now defaults
  to 24 (caller can pass body.limit up to 100), so 2-3 days of permits
  stay on the panel. Added body.contractor — passes a name into the
  WHERE so the staffer can pin a specific contractor to the panel
  ("Target Corporation" → 3 of their permits over $250K).

Server-side:
- new POST /intelligence/profiler_index — paginated contractor index
  (since, min_cost, search, limit) with merged contact_1+contact_2
  aggregations
- /intelligence/permit_contracts — body.limit + body.contractor
- /profiler and /contractors routes serve profiler.html

Front-end:
- new mcp-server/profiler.html — sortable table, live filter, deep
  links to /contractor?name=... (prefix-aware via P, so /lakehouse
  works on devop.live)
- search.html + console.html nav: added "Profiler" link

Verified end-to-end via playwright on the public URL.
This commit is contained in:
root
2026-04-27 22:00:52 -05:00
parent b02cf5b9e1
commit 08c8debfff
4 changed files with 319 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
mcp-server/index.ts
+101 -3
View File
@@ -876,6 +876,78 @@ async function main() {
}
}
// Profiler index — directory of every contractor that has filed
// a Chicago permit recently, ranked by permit count + total
// cost. Each name in the response links to the full /contractor
// profile page. Answers J's ask: "a profiler index that shows
// a history of everyone." Pulled live from Socrata; the
// count/cost aggregations let the staffer see who's actually
// active vs one-off LLCs.
if (url.pathname === "/intelligence/profiler_index" && req.method === "POST") {
const start = Date.now();
const b = await json();
const sinceDate = String(b.since || "2025-06-01");
const minCost = Math.max(0, Number(b.min_cost) || 100000);
const limit = Math.max(1, Math.min(500, Number(b.limit) || 200));
const search = String(b.search || "").trim();
const permitUrl = "https://data.cityofchicago.org/resource/ydr8-5enu.json";
// Group by contact_1_name AND by contact_2_name in two
// queries, then merge — Socrata's GROUP BY only takes one
// expression and we want both contractor slots.
const buildQuery = (col: string) => {
const where = [
`reported_cost>${minCost}`,
`issue_date>'${sinceDate.replace(/'/g, "")}'`,
`${col} IS NOT NULL`,
];
if (search) {
const s = search.replace(/'/g, "''").toUpperCase();
where.push(`upper(${col}) like '%${s}%'`);
}
return `${permitUrl}?$select=${col} AS name,count(*) as cnt,sum(reported_cost) as total_cost,max(issue_date) as last_filed&`
+ `$where=${encodeURIComponent(where.join(" AND "))}`
+ `&$group=${col}&$order=total_cost DESC&$limit=${limit}`;
};
try {
const [byC1, byC2] = await Promise.all([
fetch(buildQuery("contact_1_name")).then((r) => r.json()).catch(() => []),
fetch(buildQuery("contact_2_name")).then((r) => r.json()).catch(() => []),
]);
const merged: Record<string, { name: string; permits: number; total_cost: number; last_filed: string; roles: Set<string> }> = {};
const consume = (rows: any[], role: string) => {
for (const r of rows || []) {
const n = (r.name || "").trim();
if (!n) continue;
const cnt = parseInt(r.cnt, 10) || 0;
const cost = parseFloat(r.total_cost || "0") || 0;
const last = r.last_filed || "";
const key = n.toUpperCase();
if (!merged[key]) merged[key] = { name: n, permits: 0, total_cost: 0, last_filed: "", roles: new Set() };
merged[key].permits += cnt;
merged[key].total_cost += cost;
if (last > merged[key].last_filed) merged[key].last_filed = last;
merged[key].roles.add(role);
}
};
consume(byC1, "applicant");
consume(byC2, "contractor");
const rows = Object.values(merged)
.map((r) => ({ ...r, roles: Array.from(r.roles) }))
.sort((a, b) => b.total_cost - a.total_cost)
.slice(0, limit);
return ok({
count: rows.length,
since: sinceDate,
min_cost: minCost,
search,
contractors: rows,
duration_ms: Date.now() - start,
});
} catch (e: any) {
return err(`profiler_index: ${e.message}`, 500);
}
}
// Staffer roster — read by the UI dropdown so each coordinator
// can act under their own identity (per-staffer hot-swap index).
if (url.pathname === "/api/staffers" || url.pathname === "/staffers") {
@@ -1032,6 +1104,14 @@ async function main() {
// OSHA national, Chicago history, ticker chart, parent link,
// federal contracts, debarment, unions, training. Click any
// contractor name in a permit Entity Brief to land here.
// Profiler index — directory page of everyone who's filed a
// Chicago permit (clickable directory of contractors).
if (url.pathname === "/profiler" || url.pathname === "/contractors") {
return new Response(Bun.file(import.meta.dir + "/profiler.html"), {
headers: { "Content-Type": "text/html; charset=utf-8" },
});
}
if (url.pathname === "/contractor") {
return new Response(Bun.file(import.meta.dir + "/contractor.html"), {
headers: { ...cors, "Content-Type": "text/html" },
@@ -1351,6 +1431,7 @@ async function main() {
if (url.pathname === "/intelligence/permit_contracts" && req.method === "POST") {
const start = Date.now();
try {
const b: any = await req.json().catch(() => ({}));
const permitUrl = "https://data.cityofchicago.org/resource/ydr8-5enu.json";
// Recent + substantial permits only — skip tiny ones that
// don't imply real staffing demand.
@@ -1358,10 +1439,27 @@ async function main() {
// panel on each card can populate without a second fetch.
// Contacts identify the applicant / contractor by name —
// those are the keys we pass to OSHA/ILSOS enrichment.
// Caller-controlled limit: J reported the live panel was
// dropping older permits (Target) because $limit=6 only ever
// showed today's 6 newest. Default 24 so a few days of
// permits stay on the panel; allow up to 100 via body.
const reqLimit = Math.max(1, Math.min(100, Number((b as any)?.limit) || 24));
// Optional contractor-name filter — lets the panel scope to
// a specific contact_1 or contact_2 name (e.g. "Target
// Corporation") so the user can pin a contractor to the panel
// without it scrolling past.
const cFilter = String((b as any)?.contractor || "").trim().replace(/'/g, "''");
const wherePieces: string[] = [
"reported_cost>250000",
"issue_date>'2025-06-01'",
];
if (cFilter) {
wherePieces.push(`(upper(contact_1_name)=upper('${cFilter}') OR upper(contact_2_name)=upper('${cFilter}'))`);
}
const permits: any[] = await fetch(
`${permitUrl}?$select=id,permit_type,work_type,work_description,reported_cost,street_number,street_direction,street_name,community_area,issue_date,contact_1_name,contact_1_type,contact_2_name,contact_2_type&`
+ `$where=reported_cost>250000 AND issue_date>'2025-06-01'`
+ `&$order=issue_date DESC&$limit=6`
`${permitUrl}?$select=id,permit_type,work_type,work_description,reported_cost,street_number,street_direction,street_name,community_area,issue_date,contact_1_name,contact_1_type,contact_2_name,contact_2_type,latitude,longitude&`
+ `$where=${encodeURIComponent(wherePieces.join(" AND "))}`
+ `&$order=issue_date DESC&$limit=${reqLimit}`
).then(r => r.json()).catch(() => []);
const typeToRole: Record<string, string> = {