lakehouse

421 Commits 15 Branches 3 Tags

3 Commits

Include renames

Author SHA1 Message Date

Author	SHA1	Message	Date
rootandClaude Opus 4.8	ab1f12296d	profiler: multi-source quote aggregation with confidence scoring Replaces the single-source quote feed with a 3-source consensus system, the differentiator for the index: pull each ticker from independent feeds, take the median, and score confidence by how many corroborate it. Disagreement becomes a signal instead of a silently-wrong number. Context: Stooq retired its free q/l/ CSV quote API (404s server-side; q/d/l behind a JS challenge), which broke the profiler basket. The interim Yahoo-only fix worked but had a hidden wrong-baseline bug — Yahoo's chartPreviousClose over a 5d window references the pre-window close, so day-change read +11.65% for CAT when the real figure is +1.11%. Cross-checking against Nasdaq + CNBC exposes exactly that class of quiet error, which a single source can never self-report. - quotes.ts — Yahoo + Nasdaq + CNBC fetchers (all keyless, browser UA; Yahoo prev-close now derived from the daily candle, not the buggy meta field), median consensus, confidence = corroborating sources / 3, outlier flagging, and a bun:sqlite hourly time-series cache. - quote_ingest.ts + ops/systemd/lakehouse-quote-ingest.{service,timer} — hourly snapshot into the cache (real-time not required per product call). Runs as root to match the mcp-server that co-owns the db; busy_timeout serializes the two writers. - /intelligence/ticker_quotes — serves cache-first (fast), aggregates cold tickers on-demand once and caches them. New /quote_ingest admin trigger. - profiler.html — per-card confidence badge (green/amber/red) + a per-source breakdown tooltip; "3-source consensus" labeling. - entity.ts — the earlier Stooq->Yahoo swap for the single-ticker brief (kept fetchStooqQuote dead-but-reversible). Verified live on devop.live/lakehouse/profiler: liquid names read 100% / 3-source consensus; thin microcaps (BLPG, ESHSF — Yahoo-only) honestly read 33% / 1-source. Full basket (17 tickers) seeded; hourly timer armed. Cache db (data/_quote_cache/) is gitignored. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-18 01:28:28 -05:00
rootandClaude Opus 4.7	68d226c314	phase 1.6: BIPA withdrawal endpoint + UI + retention sweep timer Closes the four production gaps that were live after the consent endpoint shipped (`76cb5ac`): (1) Withdrawal endpoint POST /biometric/subject/{id}/withdraw backs the BIPA right of withdrawal that consent template v1 §2 explicitly promises. Without it, the only way to honor a candidate's withdrawal request was the heavier /erase, which destroys immediately rather than starting the 30-day SLA clock that the consent template commits to. Side-effects: - manifest.consent.biometric.status: Given → Withdrawn - manifest.consent.biometric.retention_until: 18mo → 30d - audit row kind=biometric_consent_withdrawal, captures reason + operator_of_record + evidence_path - DOES NOT touch general_pii or subject.status — biometric is independently revocable State machine: Given→Withdrawn (happy), NeverCollected/Pending→ 409 nothing_to_withdraw, Withdrawn→409 already_withdrawn (won't advance the destruction clock), Expired→409 already_expired, subject Erased/RetentionExpired→403 subject_inactive. 12 new unit tests covering happy path + all guards + a full grant→withdraw cycle that asserts retention_until is correctly accelerated and the audit chain has 2 rows in correct order. (2) Withdraw UI at /biometric/withdraw (mcp-server-served HTML). 3-screen flow: operator auth (token + name in sessionStorage), withdrawal form (candidate_id + free-text reason ≥10 chars + optional evidence path), confirmation showing the audit row HMAC + the 30-day retention_until clock + a curl recipe for /audit/subject/{id} verification. Same neo-brutalist styling as biometric_intake.html. Mounted at http://localhost:3700/biometric/withdraw and externally at https://devop.live/lakehouse/biometric/withdraw. (3) Retention sweep systemd timer. crates/catalogd/bin/retention_sweep binary already existed; this commit schedules it. Daily 03:00 UTC, Persistent=true so a missed boot triggers on next start. Service runs as oneshot with --apply (writes a date-stamped JSONL to data/_catalog/subjects/_retention_sweep_<date>.jsonl ONLY when overdue subjects exist, per the binary's existing semantics). install.sh updated to handle .timer + paired .service correctly: enables the timer, skips direct start of the oneshot service (the timer pulls it in). One-shot manual test confirmed clean: 100 subjects scanned, 0 overdue (all backfill subjects within their 4-year general retention window). (4) operator_of_record bug fix in intake UI. Previously the page hardcoded the literal string 'intake_ui_operator' as the operator_of_record sent to /consent — meaning every audit row captured the same useless placeholder, defeating the whole point of operator traceability. Fixed by adding an operator name field to the token-paste step (sessionStorage-backed), passed through to consent + photo POSTs as the actual operator. Verified live post-restart: - gateway /audit/health + /biometric/health both 200 - mcp-server /biometric/intake + /biometric/withdraw both 200 - Live withdraw probes: 401 (no token), 400 (empty body), 404 (ghost subject), 409 nothing_to_withdraw on WORKER-1 (which is NeverCollected per backfill default) — all expected - Binary strings contain: process_withdraw, withdraw_consent, biometric_consent_withdrawal, biometric_withdraw_response.v1, nothing_to_withdraw, already_withdrawn, already_expired, /subject/{candidate_id}/withdraw route - systemd: lakehouse-retention-sweep.timer active+enabled, next fire Tue 2026-05-05 22:00 CDT (= 03:00 UTC May 6) - Manual one-shot of retention sweep service: exit 0/SUCCESS, 100 subjects loaded, 0 overdue 83/83 catalogd lib tests + 46/46 biometric_endpoint tests green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-05 15:09:32 -05:00
profit	c85c55006d	ops: systemd units for auditor + context7 bridge lakehouse/auditor 3 warnings — see review Promotes two previously manual-start Bun services to systemd so they survive restarts + run continuously. - ops/systemd/lakehouse-auditor.service — polls Gitea every 90s, runs 4 audit checks per PR head SHA, posts commit status + review comment. Runs as root to match existing lakehouse-* service conventions on this host; can read /home/profit/.git-credentials (0600 profit:profit). - ops/systemd/lakehouse-context7-bridge.service — HTTP wrapper on :3900 for Phase 45 doc-drift detection. Decoupled from gateway; runs independently. - ops/systemd/install.sh — idempotent installer (copy → daemon-reload → enable --now). Prints post-install active/enabled status. - ops/systemd/README.md — run/stop/logs/pause docs. Pause control stays per-service (bot.paused / auditor.paused files at repo root). Not wired to branch protection yet — the auditor's commit status is currently advisory, not enforcing. Flip via Gitea branch_protections API when confident.	2026-04-22 04:15:58 -05:00

rootandClaude Opus 4.8

ab1f12296d

profiler: multi-source quote aggregation with confidence scoring

Replaces the single-source quote feed with a 3-source consensus system,
the differentiator for the index: pull each ticker from independent
feeds, take the median, and score confidence by how many corroborate it.
Disagreement becomes a signal instead of a silently-wrong number.

Context: Stooq retired its free q/l/ CSV quote API (404s server-side;
q/d/l behind a JS challenge), which broke the profiler basket. The
interim Yahoo-only fix worked but had a hidden wrong-baseline bug —
Yahoo's chartPreviousClose over a 5d window references the pre-window
close, so day-change read +11.65% for CAT when the real figure is
+1.11%. Cross-checking against Nasdaq + CNBC exposes exactly that class
of quiet error, which a single source can never self-report.

- quotes.ts — Yahoo + Nasdaq + CNBC fetchers (all keyless, browser UA;
  Yahoo prev-close now derived from the daily candle, not the buggy
  meta field), median consensus, confidence = corroborating sources / 3,
  outlier flagging, and a bun:sqlite hourly time-series cache.
- quote_ingest.ts + ops/systemd/lakehouse-quote-ingest.{service,timer} —
  hourly snapshot into the cache (real-time not required per product
  call). Runs as root to match the mcp-server that co-owns the db;
  busy_timeout serializes the two writers.
- /intelligence/ticker_quotes — serves cache-first (fast), aggregates
  cold tickers on-demand once and caches them. New /quote_ingest admin
  trigger.
- profiler.html — per-card confidence badge (green/amber/red) + a
  per-source breakdown tooltip; "3-source consensus" labeling.
- entity.ts — the earlier Stooq->Yahoo swap for the single-ticker brief
  (kept fetchStooqQuote dead-but-reversible).

Verified live on devop.live/lakehouse/profiler: liquid names read
100% / 3-source consensus; thin microcaps (BLPG, ESHSF — Yahoo-only)
honestly read 33% / 1-source. Full basket (17 tickers) seeded; hourly
timer armed. Cache db (data/_quote_cache/) is gitignored.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-18 01:28:28 -05:00

rootandClaude Opus 4.7

68d226c314

phase 1.6: BIPA withdrawal endpoint + UI + retention sweep timer

Closes the four production gaps that were live after the consent
endpoint shipped (76cb5ac):

(1) Withdrawal endpoint POST /biometric/subject/{id}/withdraw
    backs the BIPA right of withdrawal that consent template v1 §2
    explicitly promises. Without it, the only way to honor a
    candidate's withdrawal request was the heavier /erase, which
    destroys immediately rather than starting the 30-day SLA clock
    that the consent template commits to. Side-effects:
      - manifest.consent.biometric.status: Given → Withdrawn
      - manifest.consent.biometric.retention_until: 18mo → 30d
      - audit row kind=biometric_consent_withdrawal, captures
        reason + operator_of_record + evidence_path
      - DOES NOT touch general_pii or subject.status — biometric
        is independently revocable
    State machine: Given→Withdrawn (happy), NeverCollected/Pending→
    409 nothing_to_withdraw, Withdrawn→409 already_withdrawn (won't
    advance the destruction clock), Expired→409 already_expired,
    subject Erased/RetentionExpired→403 subject_inactive.
    12 new unit tests covering happy path + all guards + a full
    grant→withdraw cycle that asserts retention_until is correctly
    accelerated and the audit chain has 2 rows in correct order.

(2) Withdraw UI at /biometric/withdraw (mcp-server-served HTML).
    3-screen flow: operator auth (token + name in sessionStorage),
    withdrawal form (candidate_id + free-text reason ≥10 chars +
    optional evidence path), confirmation showing the audit row
    HMAC + the 30-day retention_until clock + a curl recipe for
    /audit/subject/{id} verification. Same neo-brutalist styling
    as biometric_intake.html. Mounted at
    http://localhost:3700/biometric/withdraw and externally at
    https://devop.live/lakehouse/biometric/withdraw.

(3) Retention sweep systemd timer. crates/catalogd/bin/retention_sweep
    binary already existed; this commit schedules it. Daily 03:00 UTC,
    Persistent=true so a missed boot triggers on next start. Service
    runs as oneshot with --apply (writes a date-stamped JSONL to
    data/_catalog/subjects/_retention_sweep_<date>.jsonl ONLY when
    overdue subjects exist, per the binary's existing semantics).
    install.sh updated to handle .timer + paired .service correctly:
    enables the timer, skips direct start of the oneshot service
    (the timer pulls it in). One-shot manual test confirmed clean:
    100 subjects scanned, 0 overdue (all backfill subjects within
    their 4-year general retention window).

(4) operator_of_record bug fix in intake UI. Previously the page
    hardcoded the literal string 'intake_ui_operator' as the
    operator_of_record sent to /consent — meaning every audit row
    captured the same useless placeholder, defeating the whole
    point of operator traceability. Fixed by adding an operator
    name field to the token-paste step (sessionStorage-backed),
    passed through to consent + photo POSTs as the actual operator.

Verified live post-restart:
- gateway /audit/health + /biometric/health both 200
- mcp-server /biometric/intake + /biometric/withdraw both 200
- Live withdraw probes: 401 (no token), 400 (empty body), 404
  (ghost subject), 409 nothing_to_withdraw on WORKER-1 (which
  is NeverCollected per backfill default) — all expected
- Binary strings contain: process_withdraw, withdraw_consent,
  biometric_consent_withdrawal, biometric_withdraw_response.v1,
  nothing_to_withdraw, already_withdrawn, already_expired,
  /subject/{candidate_id}/withdraw route
- systemd: lakehouse-retention-sweep.timer active+enabled,
  next fire Tue 2026-05-05 22:00 CDT (= 03:00 UTC May 6)
- Manual one-shot of retention sweep service: exit 0/SUCCESS,
  100 subjects loaded, 0 overdue

83/83 catalogd lib tests + 46/46 biometric_endpoint tests green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-05 15:09:32 -05:00

profit

c85c55006d

ops: systemd units for auditor + context7 bridge

lakehouse/auditor 3 warnings — see review

Promotes two previously manual-start Bun services to systemd
so they survive restarts + run continuously.

- ops/systemd/lakehouse-auditor.service — polls Gitea every 90s,
  runs 4 audit checks per PR head SHA, posts commit status + review
  comment. Runs as root to match existing lakehouse-* service
  conventions on this host; can read /home/profit/.git-credentials
  (0600 profit:profit).
- ops/systemd/lakehouse-context7-bridge.service — HTTP wrapper on
  :3900 for Phase 45 doc-drift detection. Decoupled from gateway;
  runs independently.
- ops/systemd/install.sh — idempotent installer (copy → daemon-reload
  → enable --now). Prints post-install active/enabled status.
- ops/systemd/README.md — run/stop/logs/pause docs.

Pause control stays per-service (bot.paused / auditor.paused files
at repo root). Not wired to branch protection yet — the auditor's
commit status is currently advisory, not enforcing. Flip via Gitea
branch_protections API when confident.

2026-04-22 04:15:58 -05:00