profit/lakehouse
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

profiler: multi-source quote aggregation with confidence scoring

Browse Source 
Replaces the single-source quote feed with a 3-source consensus system,
the differentiator for the index: pull each ticker from independent
feeds, take the median, and score confidence by how many corroborate it.
Disagreement becomes a signal instead of a silently-wrong number.

Context: Stooq retired its free q/l/ CSV quote API (404s server-side;
q/d/l behind a JS challenge), which broke the profiler basket. The
interim Yahoo-only fix worked but had a hidden wrong-baseline bug —
Yahoo's chartPreviousClose over a 5d window references the pre-window
close, so day-change read +11.65% for CAT when the real figure is
+1.11%. Cross-checking against Nasdaq + CNBC exposes exactly that class
of quiet error, which a single source can never self-report.

- quotes.ts — Yahoo + Nasdaq + CNBC fetchers (all keyless, browser UA;
  Yahoo prev-close now derived from the daily candle, not the buggy
  meta field), median consensus, confidence = corroborating sources / 3,
  outlier flagging, and a bun:sqlite hourly time-series cache.
- quote_ingest.ts + ops/systemd/lakehouse-quote-ingest.{service,timer} —
  hourly snapshot into the cache (real-time not required per product
  call). Runs as root to match the mcp-server that co-owns the db;
  busy_timeout serializes the two writers.
- /intelligence/ticker_quotes — serves cache-first (fast), aggregates
  cold tickers on-demand once and caches them. New /quote_ingest admin
  trigger.
- profiler.html — per-card confidence badge (green/amber/red) + a
  per-source breakdown tooltip; "3-source consensus" labeling.
- entity.ts — the earlier Stooq->Yahoo swap for the single-ticker brief
  (kept fetchStooqQuote dead-but-reversible).

Verified live on devop.live/lakehouse/profiler: liquid names read
100% / 3-source consensus; thin microcaps (BLPG, ESHSF — Yahoo-only)
honestly read 33% / 1-source. Full basket (17 tickers) seeded; hourly
timer armed. Cache db (data/_quote_cache/) is gitignored.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-06-18 01:28:28 -05:00
co-authored by Claude Opus 4.8
parent fcd53168a0
commit ab1f12296d
8 changed files with 531 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
mcp-server/index.ts
+42 -30
View File
@@ -936,45 +936,57 @@ async function main() {
}
}
// Batch ticker quotes — used by the profiler page's scrolling
// ticker basket. Stooq's HTTP CSV API is single-symbol per call,
// so this fans out N tickers in parallel and returns a flat
// map. Non-US tickers (HOC.DE, SKA-B.ST, LLC.AX) won't have a
// Stooq.us entry; we surface that as null so the UI can render
// them with a "—" placeholder.
// Batch ticker quotes — the profiler basket. Multi-source
// consensus (Yahoo + Nasdaq + CNBC), cache-first from the local
// SQLite time-series; cold tickers are aggregated live once and
// cached. Each quote carries price + day_change_pct (median across
// sources), a confidence score (corroborating sources / 3), the
// per-source breakdown, and a stale flag. Dotted/foreign symbols
// (HOC.DE, SKA-B.ST) are skipped → null → "—" placeholder.
// See quotes.ts; refreshed hourly by quote_ingest.ts.
if (url.pathname === "/intelligence/ticker_quotes" && req.method === "POST") {
const start = Date.now();
const b = await json();
const tickers: string[] = Array.isArray(b.tickers) ? b.tickers.map((t: any) => String(t).toUpperCase()).filter(Boolean) : [];
if (!tickers.length) return ok({ quotes: {}, duration_ms: 0 });
const { fetchStooqQuote } = await import("./entity.js");
const dedup = Array.from(new Set(tickers)).slice(0, 50);
const results = await Promise.all(dedup.map(async (t) => {
// Skip non-US suffixes — Stooq.us won't have them
if (t.includes(".")) return [t, null] as const;
try {
const q = await fetchStooqQuote(t);
if (!q || !q.price) return [t, null] as const;
const change_pct = q.open && q.open > 0 ? ((q.price - q.open) / q.open) * 100 : null;
return [t, {
ticker: t,
price: q.price,
price_date: q.price_date,
open: q.open,
high: q.high,
low: q.low,
day_change_pct: change_pct,
stooq_url: `https://stooq.com/q/?s=${t.toLowerCase()}.us`,
}] as const;
} catch {
return [t, null] as const;
}
}));
const { serveQuotes } = await import("./quotes.ts");
const dedup = Array.from(new Set(tickers)).slice(0, 50).filter((t) => !t.includes("."));
const served = await serveQuotes(dedup);
const quotes: Record<string, any> = {};
for (const [t, q] of results) quotes[t] = q;
for (const t of dedup) {
const a = served[t];
quotes[t] = a
? {
ticker: t,
price: a.price,
day_change_pct: a.day_change_pct,
price_date: a.price_date,
confidence: a.confidence,
source_count: a.source_count,
spread_pp: a.spread_pp,
sources: a.sources,
stale: a.stale,
as_of: a.ts,
source_url: `https://finance.yahoo.com/quote/${t}`,
}
: null;
}
return ok({ quotes, count: dedup.length, duration_ms: Date.now() - start });
}
// Manual quote-ingest trigger (admin / timer). POST {tickers?:[]}
// — with a list, ingests those; without, refreshes every tracked
// ticker. The systemd timer hits this hourly; also callable by
// hand. Returns the write summary.
if (url.pathname === "/intelligence/quote_ingest" && req.method === "POST") {
const start = Date.now();
const b = await json().catch(() => ({}));
const list = Array.isArray(b?.tickers) ? b.tickers.map((t: any) => String(t).toUpperCase()) : undefined;
const { ingestQuotes } = await import("./quotes.ts");
const r = await ingestQuotes(list);
return ok({ ...r, duration_ms: Date.now() - start });
}
// Profiler index — directory of every contractor that has filed
// a Chicago permit recently, ranked by permit count + total
// cost. Each name in the response links to the full /contractor