External callers (lakehouse observer.ts) escalate to /api/run with mode
"code_review" (underscored), but the canonical registered mode is "codereview",
so those calls hit the unknown-mode 400 and silently dropped. Normalize the alias
right after reading the mode so it validates and dispatches via RUNNERS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two hardening fixes from a fresh security pass of the live code:
- SESSION_COOKIE_SECURE=True: the session cookie set only HTTPONLY +
SAMESITE=Lax, so it could be sent over plaintext HTTP. App is served
behind nginx TLS, so no functional downside.
- _nginx_ban: validate the IP with ipaddress.ip_address() before
interpolating it into the nginx deny file. A malformed value could
otherwise inject nginx directives at the last write before reload.
Guard sits at the sink, covering all current and future callers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three issues from J 2026-04-30:
1. Silent fail2ban-client / nginx subprocess failures
Pre-fix _execute_ban called fail2ban-client with capture_output=True
but threw away the result. _nginx_ban had bare `except: pass`
swallowing everything. So a non-zero fail2ban exit (jail not
configured, IP already banned, IPv6 quirk) or PermissionError
on /etc/nginx/banned_ips.conf logged "AI_BAN" while the attacker
walked through unimpeded. The "errors in logs" J was seeing.
Now every subprocess failure surfaces:
- FAIL2BAN_FAILED rc=N stderr=... — non-zero exit
- FAIL2BAN_TIMEOUT — client didn't return in 5s
- FAIL2BAN_NOT_INSTALLED — binary missing
- NGINX_BAN_WRITE_DENIED — permission error on conf file
- NGINX_RELOAD_FAILED rc=N stderr=... — systemctl reload non-zero
- NGINX_RELOAD_TIMEOUT / NGINX_RELOAD_NO_SYSTEMCTL — runtime gaps
sec_log.error catches these so journalctl -u llm-team-ui shows
the actual reason a ban didn't stick.
2. AI auto-scan failure callback when model is busy
Pre-fix Ollama unreachable / busy / timeout silently preserved
log position + skipped the scan. Operator only learned about
the gap by manually checking sentinel-status. Now:
- 1 retry inside same scan after SENTINEL_AI_RETRY_DELAY_SECS
(30s) on connection error / timeout / 429 / 503
- 4xx errors that won't recover (404 model missing, 400 bad
prompt) fail fast without retrying
- consecutive_ai_failures counter in _sentinel_stats
- On 3+ consecutive failures, send_security_alert() fires —
"Sentinel AI unreachable" email with last error + endpoint
+ model name. One alert per outage (ai_busy_alerted flag);
clears on first successful scan so flapping doesn't spam.
- AI_RECOVERED log line on first scan after a streak.
3. Sentinel ban path still substring-matched 192.168
Same vulnerability class as admin_ban_ip had — only protected
one /16. Replaced 4 sites with is_allowlisted(ip):
- threat-list display filter (line 7638): now hides ALL
allowlisted IPs from the panel
- mass-ban API (line 8016): refuses ban for any allowlisted IP
- sentinel analysis filter (line 12786): saves AI tokens by
never sending allowlisted-IP traffic to the judge
- sentinel ban verdict gate (line 12949): defense in depth —
even if the AI says "ban" on an allowlisted IP, this catches it
Combined with the layered defenses in b09b73c (track_violation,
_auto_escalate, _nginx_ban, admin_ban_ip), there is now no
code path that can ban an allowlisted IP. Operator self-ban
is structurally impossible.
Privilege note: the systemd unit at /root/llm-team-ui/llm-team-ui.service
runs as User=root, so subprocess.run(["fail2ban-client", ...]) and
systemctl reload nginx have permission. The "errors in logs" J was
seeing weren't permission-denied; they were silent non-zero exits.
The new subprocess wrappers surface those.
If the operator later splits the app into a non-root tier
(Opus OB-3 architectural recommendation, deferred), this same
infrastructure still works — the wrappers will then surface
"PermissionError" with full path + uid context, telling the
operator exactly which command needs sudo NOPASSWD or
PolicyKit rule.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
UX request 2026-04-30: when sorting by threat in the threat intel
panel, ban-selected required clicking each per-row checkbox
individually. Pages with 20-50 threats made bulk-ban tedious.
Adds a master `[ ] all` checkbox to the toolbar (right of the
Sort buttons, left of the existing 'N selected' counter) that
toggles every per-row .ip-check on the page in one click. Then
'Ban Selected' / 'Unban Selected' work over the whole set.
Three-state: unchecked (none selected) / checked (all) /
indeterminate (partial — browsers render this as a "half-tick"
so operators get visual feedback when they've toggled some rows
manually after using master). updateSelCount keeps the master
in sync as individual rows toggle so the visual is always
truthful.
No backend change — `/api/admin/security/mass-ban` already
accepts an arbitrary IP list. This is purely a frontend
ergonomics improvement on top of the existing mass-action
infrastructure.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The OB-4 fix at 939dfdd was incomplete. It bypassed the path-regex
EXPLOIT check for allowlisted-IP admins, but _track_violation is
called from THREE sites (exploit_scan, rate_limit, login_fail) and
only the exploit_scan path had the bypass. An admin who hit a rate
limit + had a session timeout + entered a wrong password within 60s
could still self-ban via the OTHER paths, ending up locked out
through the back door.
This commit adds 4 layers of defense in depth, each independently
sufficient to stop an allowlisted IP from being banned:
1. _track_violation: bail early if is_allowlisted(ip). Allowlisted
IPs never accumulate violations from ANY path. Plus eviction
sweep when _violation_tracker grows >10K (same shape as the
_rate_limit eviction at 266de61).
2. _auto_escalate: re-check is_allowlisted before issuing any ban.
Defense in depth — if a future call path bypasses #1, this
catches it.
3. _nginx_ban: refuse to write the deny rule for allowlisted IPs,
even if a buggy caller reached this far. Last write before nginx
reload; last place to stop a bad ban.
4. admin_ban_ip: replace `ip.startswith("192.168.")` substring
check with the canonical ALLOWLIST_IPS membership test. Pre-fix
this only protected one LAN; 10.0.0.0/8, IPv6 loopback ::1, and
custom allowlist entries (e.g. an external monitoring IP) were
all banable by manual admin error. Now uses the same allowlist
as the auto-ban paths.
Operationally the admin can no longer self-ban through any path.
The auto-escalate ban audit log entries get a corresponding
"AUTO_ESCALATE_BLOCKED ip=... — allowlisted" entry instead of the
ban firing silently. Same for nginx_ban: NGINX_BAN_BLOCKED entries
log the saved bullet for operator review.
Builds on 939dfdd (OB-4 path-regex bypass) + 266de61 (rate_limit
eviction + auth_login IP gate). Together these three commits close
the LLM Team UI's ban-system self-foot-shoot vulnerability surface.
Outstanding from the scrum (architectural, separate session):
- OB-3 root-running web app + privileged shell calls
- Sentinel prompt-injection WARN
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the 2 remaining surgical-fix WARNs from the 2026-04-30
cross-lineage scrum on this codebase. OB-3 (root-running web app
with shell calls to fail2ban-client / systemctl / nginx config) and
the sentinel prompt-injection WARN both need bigger architectural
work and stay deferred.
OB-rate-limit (Opus WARN) — _rate_limit dict unbounded
Pre-fix: per-worker dict with no eviction; an attacker slowly
rotating IPs leaked memory forever. Fix: lazy eviction sweep
triggered when dict grows beyond 10K entries (cheap because we
only scan when growth is unusual). Real production wants a
Redis-backed shared counter; this is the in-process band-aid
that prevents runaway growth without changing the deploy shape.
OB-auth-setup (Opus WARN) — first-time setup grant from any IP
Pre-fix: /api/auth/login with setup=true was gated only by
COUNT(*) FROM users == 0. If the users table was ever truncated
or restored empty, the next external visitor (ANY IP) claimed
admin. Fix: also require the source IP to be in ALLOWLIST_IPS
(typically loopback + LAN gateway). Local operator setup still
works; remote attackers hitting the endpoint after an empty-
users state get 403.
Both fixes are surgical — single function, no behavior change for
the happy path. The eviction sweep runs O(n) only when n>10K and
only drops entries already past their useful window, so it never
removes an active rate-limit count.
Outstanding from the scrum (deferred):
- OB-3 root-running web app: needs split into non-root Flask tier
+ privileged sudo wrapper service. 2-4 hr architectural work.
- Sentinel prompt-injection WARN: feeds attacker-controlled UA/
path into LLM judge prompt. Needs prompt-template hardening or
output validation gate before LLM verdicts can issue ban actions.
- CSP unsafe-inline WARN: defeats most XSS protection. Removing
it requires moving inline scripts to external files (HTML
refactor).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cross-lineage scrum (Opus 4.7 + Kimi K2.6 + Qwen3-coder via the
local-review-harness chatd) on this codebase surfaced 5 BLOCK-class
issues from Opus + a convergent finding from the harness. This
commit lands the 4 surgical fixes; OB-3 (web app runs as root with
fail2ban-client + systemctl reload nginx + writes to
/etc/nginx/banned_ips.conf) needs an architectural split into a
non-root web tier + a privileged sudo wrapper, deferred for its
own session.
OB-1 — log file open at import crashes app on perm error
Pre-fix: `_sec_handler = logging.FileHandler("/var/log/llm-team-
security.log")` raised PermissionError at import time on any
non-root or fresh-install run, killing the app before Flask
started — failure was silent (no Flask process to inspect logs
on).
Fix: try/except, fall back to StreamHandler(sys.stderr) when
the path is unwritable. App starts; sec_log events still land
in journald via stderr. LLM_TEAM_SECURITY_LOG env var lets
operators override the path.
OB-2 — DB password hardcoded in source (CONVERGENT FINDING)
The `kbuser` Postgres credential
`IPbLBA0EQI8u4TeM2YZrbm1OAy5nSwqC` was leaked in source here
AND in voice-ai/audiosocket_bridge.py + voice-ai/sales_assistant.py.
Caught independently by harness LLM phase (qwen3.5 local) on
voice-ai earlier today AND Opus on this file just now. Same
password, same DB (`knowledge_base`) shared between services,
three reviewers converged.
Fix: source from LLM_TEAM_DB_DSN env var, fail loud on unset.
Operator follow-ups:
1. Rotate the password in Postgres (still in git history;
redacting source doesn't un-leak it).
2. Set LLM_TEAM_DB_DSN in /etc/llm-team-ui.env (mode 0600,
loaded via systemd EnvironmentFile=).
3. Same DSN env-var pattern needs applying to
voice-ai/audiosocket_bridge.py:47 once that branch's
workspace_context WIP lands.
OB-5 — demo_mode default=True ships public access on first boot
Pre-fix: `_demo_mode = {"active": True, ...}` + the demo branch
in login_required let users through without a session. Combined
with /api/run + /api/imagegen proxies, fresh installs were open
LLM/compute abuse surface from first boot.
Fix: default to False; LLM_TEAM_DEMO_MODE=1 env override exists
for the public devop.live deployment systemd unit so the demo
doesn't need a manual flip on every restart, but everywhere else
defaults closed.
OB-4 — EXPLOIT_PATTERNS LAN/admin lockout
Pre-fix: regex matched on `request.path` + query string against
patterns like UNION / SELECT / ;-- / <script /admin.php. Admin
URLs containing those keywords in legitimate ways (e.g. a team
name "select-rebrand" or a docs link /admin/select_a_mode) hit
3 violations in 60s and auto-banned the admin's IP. No allowlist.
Fix: bypass the path-based check for authenticated admins from
an ALLOWLIST_IPS source. Body/UA checks still apply (the prompt-
injection-as-DoS WARN in the scrum is separate). Combination
prevents self-ban without weakening the broader scanner defense.
Plus a .gitignore: /.memory/ — the local-review-harness writes
JSONL findings under <repo>/.memory/ when scanning; harness's own
gitignore is at the harness repo root, not here, so without this
the .memory/ dir would show up as untracked on every harness run
against this tree.
Other Opus WARNs deferred:
- Sentinel feeds attacker-controlled UA into LLM prompt → can
steer ban verdicts. Fix needs prompt-template hardening or
output-validation gate.
- CSP `'unsafe-inline'` defeats most XSS protection (would break
inline scripts; needs HTML refactor).
- _rate_limit unbounded dict + per-worker (needs eviction loop or
Redis-backed counter).
- auth_login first-time setup gated only by COUNT(*)==0 (needs
network-source restriction or a setup token).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New mode: Deep Analysis — 6-phase autonomous pipeline:
1. Research: all selected models answer in parallel
2. Debate: models challenge each other's findings
3. Consensus: merge research with critiques, identify strong/weak points
4. Self-Eval: structured scoring (accuracy, completeness, actionability, nuance)
5. Final Synthesis: strongest model produces definitive answer
6. Knowledge Base: result stored for future RAG retrieval
Designed for cloud models (Ollama Cloud, OpenRouter). Every successful
run trains the local knowledge base so future adaptive runs benefit.
Purple accent in mode selector to distinguish from standard modes.
Token tracking fix:
- Added est_tokens, input_chars, output_chars columns to team_runs
- save_run() now calculates and stores token estimates for ALL runs
- Both logged-in and public/demo/showcase runs track tokens
- Enables accurate usage analytics across all users
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two bugs preventing adaptive mode from working:
- ml-adaptive was missing from ML_IDS array (line 3160) — model checkboxes
never rendered, so no models were selected
- adaptive-synthesizer was missing from populateAllSelects() ids array
(line 3733) — synthesizer/judge dropdown was always empty
Both are single-line fixes. The backend pipeline (run_adaptive) was
complete and correct — self-eval, RAG retrieval, escalation, quality
scoring, KB storage all work. The UI just never wired the config.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Endpoint now returns all models, not just free ones
- Each model includes: name, context_length, free flag, prompt/completion cost
- UI shows pricing: "128K ctx · $2.50/M tok" for paid, "128K ctx · free" for free
- Filter dropdown: All Models / Free Only / Paid Only
- Search still works alongside the filter
- 29 free + 314 paid models available (GPT-5.4, Grok 4.20, Gemini 3.1, etc)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New provider: Ollama Cloud (ollama.com)
- Native Ollama chat API with bearer token auth
- Provider card in Admin → Providers tab
- "Ollama Cloud" tab with Pull Models button (fetches 36 models)
- Search/filter models, one-click Add
- Models route as ollama_cloud::modelname through query_ollama_cloud()
- Test button verifies connection
OpenRouter fix:
- Cleared bad API key from config (was dd00bea4... not sk-or-)
- Real key from /home/profit/.env now used (sk-or-v1-579...)
- Fixed OpenAI provider that had wrong base_url (ollama.com→api.openai.com)
- Bumped OR timeout to 180s for free model rate limits
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-17 23:20:44 -05:00
2 changed files with 774 additions and 66 deletions
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
profit
