Compare commits
109
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f0a3ed6832 | ||
|
|
affab8ac83 | ||
|
|
2a4b81bf48 | ||
|
|
75a0f424ef | ||
|
|
6316433062 | ||
|
|
42a11d35cd | ||
|
|
8cbbd0ef70 | ||
|
|
4cb405bb42 | ||
|
|
f44b6b3e6b | ||
|
|
5b1fcf6d27 | ||
|
|
a6f12e2609 | ||
|
|
640db8c63c | ||
|
|
138592dc56 | ||
|
|
e0a843d1a5 | ||
|
|
3fb3a60da4 | ||
|
|
52561d10d3 | ||
|
|
b95dd86556 | ||
|
|
137aed64fb | ||
|
|
ad0edbe29c | ||
|
|
5e89407939 | ||
|
|
6b71c8e9b2 | ||
|
|
a7fc8e2256 | ||
|
|
c21b261877 | ||
|
|
a663698571 | ||
|
|
330cb90f99 | ||
|
|
9c1400d738 | ||
|
|
0c4868c191 | ||
|
|
6e7ca1830e | ||
|
|
03d723e7e6 | ||
|
|
e4ae5b646e | ||
|
|
0ff091c173 | ||
|
|
af3856b103 | ||
|
|
a117ae8b38 | ||
|
|
6287558493 | ||
|
|
23eb04a145 | ||
|
|
468798c9ac | ||
|
|
76bfa2c8d7 | ||
|
|
05f2e42c45 | ||
|
|
bb1b471c67 | ||
|
|
2595d48535 | ||
|
|
1f56630d5d | ||
|
|
cdd12a1438 | ||
|
|
4aea71d213 | ||
|
|
72ee8f006f | ||
|
|
99ab0fe623 | ||
|
|
5c39c74fe4 | ||
|
|
f8e8d25b5f | ||
|
|
1274ab2cb3 | ||
|
|
95c26f04f8 | ||
|
|
20b0289aa9 | ||
|
|
25b7e6c3a7 | ||
|
|
8e3cac5812 | ||
|
|
2da8562c90 | ||
|
|
9acbe5c369 | ||
|
|
b16e485be1 | ||
|
|
bba5b826a3 | ||
|
|
df71ac7156 | ||
|
|
37804d7195 | ||
|
|
37c68d9567 | ||
|
|
e9b5498f43 | ||
|
|
be7436b6f0 | ||
|
|
e87155306b | ||
|
|
2155959013 | ||
|
|
45a95a9feb | ||
|
|
c0ff7434cb | ||
|
|
bb46869227 | ||
|
|
2279d9f51d | ||
|
|
875cfadc3d | ||
|
|
13b01fee9f | ||
|
|
845acfdcda | ||
|
|
05785b4628 | ||
|
|
7cb9999451 | ||
|
|
e7e988dcc0 | ||
|
|
5c93338f40 | ||
|
|
f9e2a0bbbe | ||
|
|
6a2cc0fb8f | ||
|
|
48c7c1c5e6 | ||
|
|
7367e5f71d | ||
|
|
66a3460c92 | ||
|
|
5aaa3c5c08 | ||
|
|
bd8c30c7bd | ||
|
|
c53d3f4d14 | ||
|
|
dd344c9b38 | ||
|
|
8d9c04a323 | ||
|
|
937569d188 | ||
|
|
1565f536eb | ||
|
|
0bd48771ff | ||
|
|
2e455919b7 | ||
|
|
8b512d30e5 | ||
|
|
25e5685f44 | ||
|
|
40305da654 | ||
|
|
cd1fda3e21 | ||
|
|
4a2bfce6e0 | ||
|
|
a001a21902 | ||
|
|
67ab6e4bac | ||
|
|
fc6b01c2bf | ||
|
|
c7e6ab3beb | ||
|
|
1bee0e4969 | ||
|
|
b532ae61f1 | ||
|
|
e1d48d3c8f | ||
|
|
546c7b081f | ||
|
|
296bdaa746 | ||
|
|
352f99de0f | ||
|
|
10383b40b7 | ||
|
|
a710896db2 | ||
|
|
f9f92706f3 | ||
|
|
b38812481e | ||
|
|
390ebf0c36 | ||
|
|
13660a017e |
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"mcpServers": {
|
||||
"lakehouse": {
|
||||
"command": "bun",
|
||||
"args": ["run", "/home/profit/lakehouse/mcp-server/index.ts"],
|
||||
"env": {
|
||||
"LAKEHOUSE_URL": "http://localhost:3100",
|
||||
"MCP_TRANSPORT": "stdio"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Generated
+743
-38
File diff suppressed because it is too large
Load Diff
+127
@@ -0,0 +1,127 @@
|
||||
# Lakehouse PR-Bot
|
||||
|
||||
A local sub-agent that reads the PRD, asks a cloud model for a small change
|
||||
proposal, applies it, runs tests, and opens a draft PR on Gitea. Manual-only
|
||||
right now — no systemd, no cron. Run one cycle at a time and watch.
|
||||
|
||||
## Run one cycle
|
||||
|
||||
```bash
|
||||
cd /home/profit/lakehouse
|
||||
bun run bot/cycle.ts
|
||||
```
|
||||
|
||||
Prerequisites:
|
||||
- Working tree must be clean (`git status` shows nothing). The bot refuses on a dirty tree so its changes don't mix with in-flight work.
|
||||
- Sidecar running on :3200, gateway on :3100, observer on :3800.
|
||||
- At least one PRD line tagged `[bot-eligible]` (see below).
|
||||
- Gitea PAT configured in `~/.git-credentials` (already set up).
|
||||
|
||||
## Tag a gap as bot-eligible
|
||||
|
||||
Add `[bot-eligible]` to any PRD line the bot is allowed to work on:
|
||||
|
||||
```md
|
||||
- [ ] Add a unit test for parse_city_state covering "South Bend, IN" edge case [bot-eligible]
|
||||
```
|
||||
|
||||
The bot scans `docs/PRD.md` for these tags. Each tagged line becomes a candidate.
|
||||
Start small — one tag at a time — until you trust the loop.
|
||||
|
||||
## Stop it mid-cycle
|
||||
|
||||
Create the pause file:
|
||||
|
||||
```bash
|
||||
touch /home/profit/lakehouse/bot.paused
|
||||
```
|
||||
|
||||
The next `bot/cycle.ts` invocation exits immediately with `skipped_pause`.
|
||||
(It does NOT kill a cycle already in-flight — use `Ctrl-C` or `pkill -f bot/cycle.ts` for that.)
|
||||
|
||||
## Budget
|
||||
|
||||
- **20 cloud calls/day, 160k tokens/day** (hard ceiling, see `bot/cost.ts`).
|
||||
- Tracked in `data/_bot/cost-YYYY-MM-DD.json`.
|
||||
- Resets at UTC midnight.
|
||||
|
||||
## Cycle outcomes
|
||||
|
||||
Every run writes a result to `data/_bot/cycles/{cycle_id}.json` and POSTs an event to the observer on :3800.
|
||||
Possible outcomes:
|
||||
|
||||
| Outcome | Meaning |
|
||||
|---|---|
|
||||
| `ok` | PR opened |
|
||||
| `cycle_noop` | proposal applied, every file was **identical to current content** (Mem0 NOOP); no test run, no PR |
|
||||
| `skipped_pause` | pause file present |
|
||||
| `skipped_cost` | daily budget exhausted |
|
||||
| `skipped_policy` | `policy.shouldRunCycle` said no |
|
||||
| `skipped_dirty_tree` | uncommitted changes present |
|
||||
| `skipped_no_gap` | no `[bot-eligible]` tags in PRD |
|
||||
| `model_failed` | sidecar/cloud model errored or returned unparseable JSON |
|
||||
| `proposal_rejected` | `policy.scoreProposal` rejected it (size, path, etc.) |
|
||||
| `apply_failed` | file write errored |
|
||||
| `tests_failed` | cargo or bun test red |
|
||||
| `pr_skipped_by_policy` | green tests, but `policy.shouldOpenPR` said no |
|
||||
| `pr_failed` | Gitea API or git push errored |
|
||||
|
||||
## Mem0-aligned apply semantics
|
||||
|
||||
`apply.ts` categorizes every proposed file into one of three modes:
|
||||
|
||||
| Mode | Trigger | Action |
|
||||
|---|---|---|
|
||||
| **ADD** | `is_new: true` and file doesn't exist | Create the file |
|
||||
| **UPDATE** | `is_new: false`, file exists, content **differs** from current | Overwrite |
|
||||
| **NOOP** | `is_new: false`, file exists, content **matches** current exactly | Skip (no write, no diff) |
|
||||
|
||||
If every file in the proposal is NOOP, the cycle short-circuits to `cycle_noop` **before** running tests or opening a PR. Mismatched shapes (`is_new:true` but file exists, or `is_new:false` but file missing) become `apply_failed` — model state confusion is surfaced, not papered over.
|
||||
|
||||
PR bodies report the three counts separately so reviewers see what actually changed vs. what was confirmed identical.
|
||||
|
||||
## How the bot compounds over cycles
|
||||
|
||||
Every finished cycle persists a `CycleResult` to `data/_bot/cycles/{cycle_id}.json`. At the **start** of the next cycle, `bot/kb.ts::loadHistory(gap_id)` scans that directory, filters to prior cycles on the same gap, and returns the five most recent outcomes (`ok`, `tests_failed`, `proposal_rejected`, `cycle_noop`, etc.) with their reasons and touched files.
|
||||
|
||||
Those outcomes are summarized into a compact block and **injected into the cloud prompt** before asking for a new proposal. The model sees things like:
|
||||
|
||||
```
|
||||
Prior attempts on this gap (3 most recent):
|
||||
- 2026-04-22 03:15 UTC — tests_failed
|
||||
reason: cargo test -p vectord::lance failed on field_type_coerce
|
||||
- 2026-04-22 02:48 UTC — proposal_rejected
|
||||
reason: touched forbidden path (docs/ADR-): docs/ADR-019-update.md
|
||||
- 2026-04-22 01:23 UTC — ok PR: https://git.agentview.dev/profit/lakehouse/pulls/142
|
||||
reason: PR #142 opened
|
||||
|
||||
Learn from these: build on what worked, avoid paths that failed.
|
||||
```
|
||||
|
||||
This is the same compounding pattern `scenario.ts` uses via `kb.loadRecommendation` — the bot's cycles are the bot's memory. No embedding, no separate jsonl, no cross-run orchestration required. First cycle on a new gap skips the block cleanly.
|
||||
|
||||
The observer events POSTed on every cycle carry the same data, so `GET :3800/stats?source=bot` aggregates "% of bot cycles on similar gaps that landed a PR" without extra plumbing.
|
||||
|
||||
## Where YOU edit
|
||||
|
||||
`bot/policy.ts` — four small functions that define what the bot does. The rest
|
||||
(`propose.ts`, `apply.ts`, `test.ts`, `pr.ts`, `cycle.ts`, `kb.ts`) is mechanical
|
||||
orchestration — you shouldn't need to touch it unless a pipeline changes.
|
||||
|
||||
One policy upgrade the history opens up: in `shouldRunCycle`, you can now bail
|
||||
out if the same gap has failed N times in a row. Example addition:
|
||||
|
||||
```ts
|
||||
import { loadHistory, statsFor } from "./kb.ts";
|
||||
// inside shouldRunCycle — but you'd need the gap, which is picked later.
|
||||
// A more natural place is scoreProposal: if prior N failures on this
|
||||
// gap's path+summary pattern, reject before testing.
|
||||
```
|
||||
|
||||
## Hard-coded guardrails (not in policy — can't be disabled)
|
||||
|
||||
- Bot **never deletes files** (`apply.ts` has no delete path).
|
||||
- Bot **never touches** `.git/`, `secrets`, `lakehouse.toml`, `docs/ADR-*`, `docs/DECISIONS.md`, `docs/PRD.md`, `/etc/`, `/root/`, `Cargo.lock`.
|
||||
- All paths validated for traversal (`path/../`) and repo-escape.
|
||||
- PRs always open as **draft** — never auto-merge.
|
||||
- Budget check is before the policy function runs — no way to override the daily cap from `policy.ts`.
|
||||
@@ -0,0 +1,86 @@
|
||||
// Apply a proposal to disk with Mem0-aligned ADD / UPDATE / NOOP
|
||||
// semantics. Whole-file writes only — never deletes. All paths
|
||||
// validated repo-relative + path-traversal-free.
|
||||
//
|
||||
// Three outcomes per file:
|
||||
// ADD — file didn't exist, new content written (is_new must be true)
|
||||
// UPDATE — file existed, content differs from current, overwritten
|
||||
// NOOP — file existed, content identical to current, nothing written
|
||||
//
|
||||
// The NOOP case is important: a cloud proposal that "re-derives" an
|
||||
// existing file shouldn't churn the repo or burn test cycles.
|
||||
|
||||
import { writeFile, readFile, mkdir, access } from "node:fs/promises";
|
||||
import { dirname, resolve, relative } from "node:path";
|
||||
import type { ApplyOutcome, Proposal } from "./types.ts";
|
||||
|
||||
const REPO_ROOT = "/home/profit/lakehouse";
|
||||
|
||||
export async function applyProposal(p: Proposal): Promise<ApplyOutcome> {
|
||||
const added: string[] = [];
|
||||
const updated: string[] = [];
|
||||
const noop: string[] = [];
|
||||
const errors: string[] = [];
|
||||
|
||||
for (const f of p.files) {
|
||||
const err = validatePath(f.path);
|
||||
if (err) {
|
||||
errors.push(`${f.path}: ${err}`);
|
||||
continue;
|
||||
}
|
||||
const abs = resolve(REPO_ROOT, f.path);
|
||||
try {
|
||||
const exists = await fileExists(abs);
|
||||
|
||||
if (f.is_new && exists) {
|
||||
errors.push(`${f.path}: is_new=true but file exists`);
|
||||
continue;
|
||||
}
|
||||
if (!f.is_new && !exists) {
|
||||
// Proposal claims to update a file that doesn't exist.
|
||||
// Refuse rather than silently ADD — model is confused about
|
||||
// repo state, and we want that surfaced, not papered over.
|
||||
errors.push(`${f.path}: is_new=false but file does not exist`);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!f.is_new) {
|
||||
// UPDATE candidate — compare to detect NOOP.
|
||||
const current = await readFile(abs, "utf8");
|
||||
if (current === f.content) {
|
||||
noop.push(f.path);
|
||||
continue;
|
||||
}
|
||||
await writeFile(abs, f.content);
|
||||
updated.push(f.path);
|
||||
} else {
|
||||
// ADD — file doesn't exist yet.
|
||||
await mkdir(dirname(abs), { recursive: true });
|
||||
await writeFile(abs, f.content);
|
||||
added.push(f.path);
|
||||
}
|
||||
} catch (e) {
|
||||
errors.push(`${f.path}: ${(e as Error).message}`);
|
||||
}
|
||||
}
|
||||
return { added, updated, noop, errors };
|
||||
}
|
||||
|
||||
function validatePath(p: string): string | null {
|
||||
if (!p) return "empty path";
|
||||
if (p.startsWith("/")) return "must be repo-relative";
|
||||
if (p.startsWith("..") || p.includes("/../")) return "path traversal";
|
||||
const abs = resolve(REPO_ROOT, p);
|
||||
const rel = relative(REPO_ROOT, abs);
|
||||
if (rel.startsWith("..")) return "resolves outside repo";
|
||||
return null;
|
||||
}
|
||||
|
||||
async function fileExists(abs: string): Promise<boolean> {
|
||||
try {
|
||||
await access(abs);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
+52
@@ -0,0 +1,52 @@
|
||||
// Daily cost tracker. Resets at UTC midnight by keying the file by date.
|
||||
// Hard ceilings are the policy surface — the bot refuses to call the
|
||||
// cloud once the budget is exhausted for the day.
|
||||
|
||||
import { readFile, writeFile, mkdir } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import type { CostState } from "./types.ts";
|
||||
|
||||
const COST_DIR = "/home/profit/lakehouse/data/_bot";
|
||||
|
||||
export const DAILY_CALLS_BUDGET = 20;
|
||||
export const DAILY_TOKENS_BUDGET = 8000 * DAILY_CALLS_BUDGET; // 160k tokens/day ceiling
|
||||
|
||||
function todayUtc(): string {
|
||||
return new Date().toISOString().slice(0, 10);
|
||||
}
|
||||
|
||||
function costFile(date: string): string {
|
||||
return join(COST_DIR, `cost-${date}.json`);
|
||||
}
|
||||
|
||||
export async function readCost(): Promise<CostState> {
|
||||
const date = todayUtc();
|
||||
try {
|
||||
const raw = await readFile(costFile(date), "utf8");
|
||||
return JSON.parse(raw) as CostState;
|
||||
} catch {
|
||||
return { date, calls: 0, tokens: 0 };
|
||||
}
|
||||
}
|
||||
|
||||
export async function recordCost(calls: number, tokens: number): Promise<CostState> {
|
||||
await mkdir(COST_DIR, { recursive: true });
|
||||
const current = await readCost();
|
||||
const updated: CostState = {
|
||||
date: current.date,
|
||||
calls: current.calls + calls,
|
||||
tokens: current.tokens + tokens,
|
||||
};
|
||||
await writeFile(costFile(updated.date), JSON.stringify(updated, null, 2));
|
||||
return updated;
|
||||
}
|
||||
|
||||
export function budgetCheck(c: CostState): { ok: boolean; reason: string } {
|
||||
if (c.calls >= DAILY_CALLS_BUDGET) {
|
||||
return { ok: false, reason: `daily call budget exhausted (${c.calls}/${DAILY_CALLS_BUDGET})` };
|
||||
}
|
||||
if (c.tokens >= DAILY_TOKENS_BUDGET) {
|
||||
return { ok: false, reason: `daily token budget exhausted (${c.tokens}/${DAILY_TOKENS_BUDGET})` };
|
||||
}
|
||||
return { ok: true, reason: `${c.calls}/${DAILY_CALLS_BUDGET} calls, ${c.tokens}/${DAILY_TOKENS_BUDGET} tokens used today` };
|
||||
}
|
||||
+345
@@ -0,0 +1,345 @@
|
||||
// Single-cycle orchestrator. This is the manual entry point:
|
||||
// bun run bot/cycle.ts
|
||||
//
|
||||
// Steps (each step can short-circuit with a clear outcome):
|
||||
// 1. Pause-file check → skipped_pause
|
||||
// 2. Cost budget check → skipped_cost
|
||||
// 3. Gather CycleContext (dirty tree, last cycle, autotune status)
|
||||
// 4. policy.shouldRunCycle → skipped_policy
|
||||
// 5. findGaps + policy.pickGap → skipped_no_gap
|
||||
// 6. generateProposal → model_failed
|
||||
// 7. policy.scoreProposal → proposal_rejected
|
||||
// 8. git checkout new branch
|
||||
// 9. applyProposal → apply_failed
|
||||
// 10. runTests → tests_failed
|
||||
// 11. policy.shouldOpenPR → pr_skipped_by_policy
|
||||
// 12. git commit + push
|
||||
// 13. openPr → pr_failed / ok
|
||||
// 14. recordCost + postEvent (every outcome)
|
||||
// 15. write cycle result to data/_bot/cycles/{cycle_id}.json
|
||||
|
||||
import { readFile, writeFile, mkdir, access } from "node:fs/promises";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { join } from "node:path";
|
||||
|
||||
import type { CycleContext, CycleResult, CycleOutcome, Proposal } from "./types.ts";
|
||||
import * as policy from "./policy.ts";
|
||||
import { readCost, recordCost, budgetCheck, DAILY_CALLS_BUDGET, DAILY_TOKENS_BUDGET } from "./cost.ts";
|
||||
import { findGaps, generateProposal } from "./propose.ts";
|
||||
import { applyProposal } from "./apply.ts";
|
||||
import { runTests } from "./test.ts";
|
||||
import { openPr } from "./pr.ts";
|
||||
import { postEvent } from "./observer.ts";
|
||||
import { loadHistory, summarizeHistory, statsFor } from "./kb.ts";
|
||||
|
||||
const REPO_ROOT = "/home/profit/lakehouse";
|
||||
const PAUSE_FILE = `${REPO_ROOT}/bot.paused`;
|
||||
const CYCLES_DIR = `${REPO_ROOT}/data/_bot/cycles`;
|
||||
const GATEWAY_URL = process.env.LH_GATEWAY_URL ?? "http://localhost:3100";
|
||||
|
||||
function log(msg: string) { console.log(`[bot] ${msg}`); }
|
||||
|
||||
async function fileExists(p: string): Promise<boolean> {
|
||||
try { await access(p); return true; } catch { return false; }
|
||||
}
|
||||
|
||||
function git(args: string[]): { code: number; stdout: string; stderr: string } {
|
||||
const r = spawnSync("git", args, { cwd: REPO_ROOT, encoding: "utf8" });
|
||||
return { code: r.status ?? -1, stdout: r.stdout ?? "", stderr: r.stderr ?? "" };
|
||||
}
|
||||
|
||||
async function gatherContext(cost: { calls: number; tokens: number }): Promise<CycleContext> {
|
||||
// Last cycle: newest file in CYCLES_DIR.
|
||||
let lastCycleAt: string | null = null;
|
||||
let lastCycleGapId: string | null = null;
|
||||
try {
|
||||
const { readdir, stat } = await import("node:fs/promises");
|
||||
const entries = await readdir(CYCLES_DIR).catch(() => [] as string[]);
|
||||
let newestTs = 0;
|
||||
let newestPath: string | null = null;
|
||||
for (const e of entries) {
|
||||
if (!e.endsWith(".json")) continue;
|
||||
const p = join(CYCLES_DIR, e);
|
||||
const s = await stat(p);
|
||||
if (s.mtimeMs > newestTs) { newestTs = s.mtimeMs; newestPath = p; }
|
||||
}
|
||||
if (newestPath) {
|
||||
const r = JSON.parse(await readFile(newestPath, "utf8")) as CycleResult;
|
||||
lastCycleAt = r.started_at;
|
||||
lastCycleGapId = r.gap?.id ?? null;
|
||||
}
|
||||
} catch {}
|
||||
|
||||
// Autotune status — if the agent is busy, skip this cycle to avoid
|
||||
// contention on shared Ollama GPU memory.
|
||||
let autotuneBusy = false;
|
||||
try {
|
||||
const r = await fetch(`${GATEWAY_URL}/vectors/agent/status`, { signal: AbortSignal.timeout(2000) });
|
||||
if (r.ok) {
|
||||
const j = await r.json() as any;
|
||||
autotuneBusy = Boolean(j.running) && (j.queue_depth ?? 0) > 0;
|
||||
}
|
||||
} catch {}
|
||||
|
||||
// Dirty tree check.
|
||||
const status = git(["status", "--porcelain"]);
|
||||
const workingTreeDirty = status.code === 0 && status.stdout.trim().length > 0;
|
||||
|
||||
return {
|
||||
startedAt: new Date().toISOString(),
|
||||
dailyCallsUsed: cost.calls,
|
||||
dailyCallsBudget: DAILY_CALLS_BUDGET,
|
||||
dailyTokensUsed: cost.tokens,
|
||||
dailyTokensBudget: DAILY_TOKENS_BUDGET,
|
||||
lastCycleAt,
|
||||
lastCycleGapId,
|
||||
autotuneBusy,
|
||||
workingTreeDirty,
|
||||
};
|
||||
}
|
||||
|
||||
async function persistResult(res: CycleResult): Promise<void> {
|
||||
await mkdir(CYCLES_DIR, { recursive: true });
|
||||
await writeFile(join(CYCLES_DIR, `${res.cycle_id}.json`), JSON.stringify(res, null, 2));
|
||||
}
|
||||
|
||||
function sigHash(gap_id: string | null, proposalSummary: string): string {
|
||||
const h = createHash("sha256");
|
||||
h.update(gap_id ?? "no-gap");
|
||||
h.update("|");
|
||||
h.update(proposalSummary);
|
||||
return h.digest("hex").slice(0, 16);
|
||||
}
|
||||
|
||||
function emit(res: CycleResult): Promise<void> {
|
||||
return postEvent({
|
||||
source: "bot",
|
||||
cycle_id: res.cycle_id,
|
||||
sig_hash: sigHash(res.gap?.id ?? null, res.proposal?.summary ?? ""),
|
||||
event_kind: res.outcome,
|
||||
ok: res.outcome === "ok",
|
||||
staffer_id: res.proposal?.model_used,
|
||||
turns: res.cloud_calls,
|
||||
duration_ms: new Date(res.ended_at).getTime() - new Date(res.started_at).getTime(),
|
||||
extra: {
|
||||
reason: res.reason,
|
||||
pr_url: res.prUrl,
|
||||
files_added: res.filesAdded,
|
||||
files_updated: res.filesUpdated,
|
||||
files_noop: res.filesNoop,
|
||||
tests_green: res.testsGreen,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const cycle_id = `bot-${new Date().toISOString().replace(/[:.]/g, "-")}-${randomUUID().slice(0, 8)}`;
|
||||
const started_at = new Date().toISOString();
|
||||
log(`cycle ${cycle_id} starting`);
|
||||
|
||||
const finish = async (outcome: CycleOutcome, reason: string, extra: Partial<CycleResult> = {}): Promise<CycleResult> => {
|
||||
const res: CycleResult = {
|
||||
cycle_id,
|
||||
started_at,
|
||||
ended_at: new Date().toISOString(),
|
||||
outcome,
|
||||
reason,
|
||||
gap: null,
|
||||
proposal: null,
|
||||
filesAdded: [],
|
||||
filesUpdated: [],
|
||||
filesNoop: [],
|
||||
testsGreen: null,
|
||||
testsOutput: "",
|
||||
prUrl: null,
|
||||
tokens_used: 0,
|
||||
cloud_calls: 0,
|
||||
...extra,
|
||||
};
|
||||
await persistResult(res);
|
||||
await emit(res);
|
||||
log(`→ ${outcome}: ${reason}`);
|
||||
return res;
|
||||
};
|
||||
|
||||
// 1. Pause.
|
||||
if (await fileExists(PAUSE_FILE)) {
|
||||
return finish("skipped_pause", `pause file at ${PAUSE_FILE}`);
|
||||
}
|
||||
|
||||
// 2. Cost.
|
||||
const cost = await readCost();
|
||||
const bc = budgetCheck(cost);
|
||||
if (!bc.ok) return finish("skipped_cost", bc.reason);
|
||||
log(`cost: ${bc.reason}`);
|
||||
|
||||
// 3+4. Context + policy.shouldRunCycle.
|
||||
const ctx = await gatherContext(cost);
|
||||
const srq = policy.shouldRunCycle(ctx);
|
||||
if (!srq.run) {
|
||||
return finish(ctx.workingTreeDirty ? "skipped_dirty_tree" : "skipped_policy", srq.reason);
|
||||
}
|
||||
|
||||
// 5. Gaps + policy.pickGap.
|
||||
const gaps = await findGaps();
|
||||
log(`found ${gaps.length} [bot-eligible] gap(s) in PRD`);
|
||||
const gapPick = policy.pickGap(gaps);
|
||||
if (!gapPick.gap) return finish("skipped_no_gap", gapPick.reason);
|
||||
const gap = gapPick.gap;
|
||||
log(`picked gap ${gap.id}: "${gap.prd_line.slice(0, 80)}"`);
|
||||
|
||||
// 6. KB lookup + Proposal.
|
||||
// Bot cycles compound: prior cycles on this gap inform the current
|
||||
// proposal. Summary is a compact block injected into the cloud prompt
|
||||
// so the model can build on past successes and steer around past
|
||||
// failures. Empty string when this is the first cycle on this gap.
|
||||
const history = await loadHistory(gap.id, 5);
|
||||
const stats = statsFor(history);
|
||||
if (history.length > 0) {
|
||||
log(`kb: ${stats.attempts} prior cycle(s) — ${stats.pr_opened} pr / ${stats.tests_failed} tests_failed / ${stats.proposal_rejected} rejected / ${stats.noop} noop`);
|
||||
} else {
|
||||
log(`kb: first cycle on this gap`);
|
||||
}
|
||||
const historySummary = summarizeHistory(history);
|
||||
|
||||
let proposal: Proposal;
|
||||
try {
|
||||
proposal = await generateProposal(gap, historySummary);
|
||||
log(`proposal: ${proposal.summary} (${proposal.files.length} files, ~${proposal.estimated_loc} LOC, ${proposal.tokens_used} tokens)`);
|
||||
} catch (e) {
|
||||
await recordCost(1, 0);
|
||||
return finish("model_failed", (e as Error).message, { gap, cloud_calls: 1 });
|
||||
}
|
||||
await recordCost(1, proposal.tokens_used);
|
||||
|
||||
// 7. Score.
|
||||
const score = policy.scoreProposal(proposal);
|
||||
if (!score.accept) {
|
||||
return finish("proposal_rejected", score.reason, { gap, proposal, tokens_used: proposal.tokens_used, cloud_calls: 1 });
|
||||
}
|
||||
log(`proposal accepted: ${score.reason}`);
|
||||
|
||||
// 8. Branch.
|
||||
const dayStamp = new Date().toISOString().slice(0, 10).replace(/-/g, "");
|
||||
const branch = `bot/cycle-${dayStamp}-${gap.id}`;
|
||||
// Branch must be fresh. If one already exists from a prior failed cycle
|
||||
// for the same gap, bail — don't silently overwrite or reuse stale state.
|
||||
const check = git(["rev-parse", "--verify", `refs/heads/${branch}`]);
|
||||
if (check.code === 0) {
|
||||
return finish("apply_failed",
|
||||
`branch ${branch} already exists — delete it manually if you want a fresh cycle for this gap`,
|
||||
{ gap, proposal, tokens_used: proposal.tokens_used, cloud_calls: 1 });
|
||||
}
|
||||
const co = git(["checkout", "-b", branch]);
|
||||
if (co.code !== 0) {
|
||||
return finish("apply_failed", `git checkout -b failed: ${co.stderr}`,
|
||||
{ gap, proposal, tokens_used: proposal.tokens_used, cloud_calls: 1 });
|
||||
}
|
||||
log(`on branch ${branch}`);
|
||||
|
||||
// 9. Apply — Mem0 ADD/UPDATE/NOOP semantics.
|
||||
const ap = await applyProposal(proposal);
|
||||
if (ap.errors.length > 0) {
|
||||
git(["checkout", "main"]);
|
||||
git(["branch", "-D", branch]);
|
||||
return finish("apply_failed", ap.errors.join("; "),
|
||||
{ gap, proposal, tokens_used: proposal.tokens_used, cloud_calls: 1 });
|
||||
}
|
||||
const changed = [...ap.added, ...ap.updated];
|
||||
log(`apply: ${ap.added.length} add, ${ap.updated.length} update, ${ap.noop.length} noop`);
|
||||
|
||||
// Short-circuit: if every proposed file was identical to what's on
|
||||
// disk, there's nothing to test or PR. Clean up the branch and exit.
|
||||
if (changed.length === 0) {
|
||||
git(["checkout", "main"]);
|
||||
git(["branch", "-D", branch]);
|
||||
return finish("cycle_noop",
|
||||
`all ${proposal.files.length} proposed file(s) were identical to current state`,
|
||||
{ gap, proposal, filesNoop: ap.noop,
|
||||
tokens_used: proposal.tokens_used, cloud_calls: 1 });
|
||||
}
|
||||
|
||||
// 10. Test.
|
||||
log(`running tests (this will take a while)...`);
|
||||
const testRes = await runTests();
|
||||
if (!testRes.green) {
|
||||
git(["checkout", "."]);
|
||||
git(["checkout", "main"]);
|
||||
git(["branch", "-D", branch]);
|
||||
return finish("tests_failed", "cargo or bun test failed", {
|
||||
gap, proposal,
|
||||
filesAdded: ap.added, filesUpdated: ap.updated, filesNoop: ap.noop,
|
||||
testsGreen: false, testsOutput: testRes.output,
|
||||
tokens_used: proposal.tokens_used, cloud_calls: 1,
|
||||
});
|
||||
}
|
||||
log(`tests green`);
|
||||
|
||||
// 11. PR gate.
|
||||
const partial: CycleResult = {
|
||||
cycle_id, started_at,
|
||||
ended_at: new Date().toISOString(),
|
||||
outcome: "ok", reason: "",
|
||||
gap, proposal,
|
||||
filesAdded: ap.added, filesUpdated: ap.updated, filesNoop: ap.noop,
|
||||
testsGreen: true, testsOutput: testRes.output,
|
||||
prUrl: null, tokens_used: proposal.tokens_used, cloud_calls: 1,
|
||||
};
|
||||
const prGate = policy.shouldOpenPR(partial);
|
||||
if (!prGate.open) {
|
||||
git(["checkout", "main"]);
|
||||
git(["branch", "-D", branch]);
|
||||
return finish("pr_skipped_by_policy", prGate.reason, partial);
|
||||
}
|
||||
|
||||
// 12. Commit + push.
|
||||
git(["add", ...changed]);
|
||||
const commitMsg = [
|
||||
`bot: ${proposal.summary}`,
|
||||
"",
|
||||
proposal.rationale,
|
||||
"",
|
||||
`PRD gap: ${gap.prd_line}`,
|
||||
`Cycle: ${cycle_id}`,
|
||||
"",
|
||||
`Co-Authored-By: ${proposal.model_used} (cloud) <bot@lakehouse.local>`,
|
||||
].join("\n");
|
||||
const ci = git(["commit", "-m", commitMsg]);
|
||||
if (ci.code !== 0) {
|
||||
return finish("pr_failed", `git commit failed: ${ci.stderr}`, partial);
|
||||
}
|
||||
const ps = git(["push", "-u", "origin", branch]);
|
||||
if (ps.code !== 0) {
|
||||
return finish("pr_failed", `git push failed: ${ps.stderr}`, partial);
|
||||
}
|
||||
|
||||
// 13. Open PR.
|
||||
try {
|
||||
const pr = await openPr({
|
||||
branch,
|
||||
title: `bot: ${proposal.summary}`,
|
||||
body: [
|
||||
`**Gap** (PRD line ${gap.line_number}): ${gap.prd_line}`,
|
||||
"",
|
||||
`**Rationale**`,
|
||||
proposal.rationale,
|
||||
"",
|
||||
`**Cycle**: \`${cycle_id}\``,
|
||||
`**Model**: \`${proposal.model_used}\``,
|
||||
`**Tokens**: ${proposal.tokens_used}`,
|
||||
`**Added**: ${ap.added.map(f => `\`${f}\``).join(", ") || "_none_"}`,
|
||||
`**Updated**: ${ap.updated.map(f => `\`${f}\``).join(", ") || "_none_"}`,
|
||||
ap.noop.length > 0 ? `**NOOP (identical to current)**: ${ap.noop.map(f => `\`${f}\``).join(", ")}` : "",
|
||||
].join("\n"),
|
||||
});
|
||||
return finish("ok", `PR #${pr.number} opened`, { ...partial, prUrl: pr.html_url });
|
||||
} catch (e) {
|
||||
return finish("pr_failed", (e as Error).message, partial);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch(e => {
|
||||
console.error("[bot] fatal:", e);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -0,0 +1,101 @@
|
||||
// Bot-local knowledge base. Every finished cycle already persists a
|
||||
// CycleResult to data/_bot/cycles/{id}.json — that IS the outcome log.
|
||||
// KB here just reads that dir, filters to prior cycles on the same gap,
|
||||
// and produces a short summary the cloud model can condition on.
|
||||
//
|
||||
// No separate jsonl, no new write path, no embedding calls. The bot's
|
||||
// "memory" is the same primary artifact that the observer consumes.
|
||||
//
|
||||
// Future: embedding-based neighbor matching across gaps (cheap once
|
||||
// sidecar is local), cross-pollination with scenario KB's
|
||||
// pathway_recommendations. Not required for the feedback loop to work
|
||||
// on a single gap — that's the floor we're building first.
|
||||
|
||||
import { readdir, readFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import type { CycleResult } from "./types.ts";
|
||||
|
||||
const CYCLES_DIR = "/home/profit/lakehouse/data/_bot/cycles";
|
||||
|
||||
export interface HistoryEntry {
|
||||
cycle_id: string;
|
||||
ended_at: string;
|
||||
outcome: string;
|
||||
reason: string;
|
||||
pr_url: string | null;
|
||||
tests_green: boolean | null;
|
||||
files_added: string[];
|
||||
files_updated: string[];
|
||||
tokens_used: number;
|
||||
}
|
||||
|
||||
export async function loadHistory(gap_id: string, max: number = 5): Promise<HistoryEntry[]> {
|
||||
let entries: string[] = [];
|
||||
try {
|
||||
entries = await readdir(CYCLES_DIR);
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
const matches: HistoryEntry[] = [];
|
||||
for (const e of entries) {
|
||||
if (!e.endsWith(".json")) continue;
|
||||
try {
|
||||
const raw = await readFile(join(CYCLES_DIR, e), "utf8");
|
||||
const r = JSON.parse(raw) as CycleResult;
|
||||
if (r.gap?.id !== gap_id) continue;
|
||||
matches.push({
|
||||
cycle_id: r.cycle_id,
|
||||
ended_at: r.ended_at,
|
||||
outcome: r.outcome,
|
||||
reason: r.reason,
|
||||
pr_url: r.prUrl,
|
||||
tests_green: r.testsGreen,
|
||||
files_added: r.filesAdded ?? [],
|
||||
files_updated: r.filesUpdated ?? [],
|
||||
tokens_used: r.tokens_used,
|
||||
});
|
||||
} catch {
|
||||
// Skip unreadable / malformed cycle files. Don't fail the current
|
||||
// cycle because an old one is corrupt.
|
||||
}
|
||||
}
|
||||
matches.sort((a, b) => b.ended_at.localeCompare(a.ended_at));
|
||||
return matches.slice(0, max);
|
||||
}
|
||||
|
||||
// Compact prompt-ready summary. Empty string when there's no history —
|
||||
// caller can skip the "prior attempts" block entirely.
|
||||
export function summarizeHistory(h: HistoryEntry[]): string {
|
||||
if (h.length === 0) return "";
|
||||
const lines = h.map(e => {
|
||||
const when = e.ended_at.slice(0, 16).replace("T", " ");
|
||||
const files = [...e.files_added, ...e.files_updated];
|
||||
const filesStr = files.length > 0 ? ` touched: ${files.join(", ")}` : "";
|
||||
const prStr = e.pr_url ? ` PR: ${e.pr_url}` : "";
|
||||
return `- ${when} UTC — ${e.outcome}${prStr}${filesStr}\n reason: ${e.reason}`;
|
||||
});
|
||||
return [
|
||||
`Prior attempts on this gap (${h.length} most recent):`,
|
||||
...lines,
|
||||
"",
|
||||
"Learn from these: build on what worked, avoid paths that failed.",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
// Aggregate stats for telemetry — lets the bot expose "% of cycles on
|
||||
// this gap that landed a PR" without re-parsing the raw history.
|
||||
export function statsFor(h: HistoryEntry[]): {
|
||||
attempts: number;
|
||||
pr_opened: number;
|
||||
tests_failed: number;
|
||||
proposal_rejected: number;
|
||||
noop: number;
|
||||
} {
|
||||
return {
|
||||
attempts: h.length,
|
||||
pr_opened: h.filter(e => e.outcome === "ok").length,
|
||||
tests_failed: h.filter(e => e.outcome === "tests_failed").length,
|
||||
proposal_rejected: h.filter(e => e.outcome === "proposal_rejected").length,
|
||||
noop: h.filter(e => e.outcome === "cycle_noop").length,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
// POST bot cycle outcomes to the observer on :3800 so they accumulate
|
||||
// in the KB alongside scenario + autotune events. Non-fatal on failure:
|
||||
// the cycle result is also written to disk so observability isn't a
|
||||
// single point of failure.
|
||||
|
||||
const OBSERVER_URL = process.env.LH_OBSERVER_URL ?? "http://localhost:3800";
|
||||
|
||||
export interface BotObserverEvent {
|
||||
source: "bot";
|
||||
cycle_id: string;
|
||||
sig_hash: string; // stable hash of the gap + proposal for dedup
|
||||
event_kind: string; // cycle outcome (ok, tests_failed, ...)
|
||||
ok: boolean;
|
||||
staffer_id?: string; // the model used, e.g. "gpt-oss:120b"
|
||||
turns?: number; // number of cloud calls this cycle
|
||||
duration_ms?: number;
|
||||
extra?: Record<string, any>;
|
||||
}
|
||||
|
||||
export async function postEvent(ev: BotObserverEvent): Promise<void> {
|
||||
try {
|
||||
const r = await fetch(`${OBSERVER_URL}/event`, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify(ev),
|
||||
signal: AbortSignal.timeout(3000),
|
||||
});
|
||||
if (!r.ok) {
|
||||
console.error(`[bot/observer] ${r.status}: ${await r.text()}`);
|
||||
}
|
||||
} catch (e) {
|
||||
console.error(`[bot/observer] POST failed: ${(e as Error).message}`);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
// ═══════════════════════════════════════════════════════════════════
|
||||
// YOU WRITE THIS FILE. These four functions shape the bot's entire
|
||||
// behavior. Defaults below are safe and intentionally conservative —
|
||||
// redline to taste. Each function is ~5-10 lines.
|
||||
// ═══════════════════════════════════════════════════════════════════
|
||||
|
||||
import type { CycleContext, CycleResult, Gap, Proposal } from "./types.ts";
|
||||
|
||||
// Should this cycle even fire right now?
|
||||
// Called AFTER the pause-file check + cost-cap check have already passed.
|
||||
// Use this for judgment calls (cooldown, shared-resource conflicts).
|
||||
export function shouldRunCycle(ctx: CycleContext): { run: boolean; reason: string } {
|
||||
if (ctx.workingTreeDirty) {
|
||||
return { run: false, reason: "dirty working tree — commit or stash your work first" };
|
||||
}
|
||||
if (ctx.lastCycleAt) {
|
||||
const minsAgo = (Date.now() - new Date(ctx.lastCycleAt).getTime()) / 60000;
|
||||
if (minsAgo < 10) return { run: false, reason: `last cycle ${minsAgo.toFixed(1)}min ago (<10min cooldown)` };
|
||||
}
|
||||
if (ctx.autotuneBusy) {
|
||||
return { run: false, reason: "autotune agent is running — avoid Ollama contention" };
|
||||
}
|
||||
return { run: true, reason: "ready" };
|
||||
}
|
||||
|
||||
// Which [bot-eligible] gap from the PRD does this cycle attack?
|
||||
// Default: first in document order. Alternatives: random for exploration,
|
||||
// smallest-context, tag-priority (e.g. [bot-eligible:high] wins).
|
||||
export function pickGap(gaps: Gap[]): { gap: Gap | null; reason: string } {
|
||||
if (gaps.length === 0) return { gap: null, reason: "no [bot-eligible] gaps in PRD" };
|
||||
return { gap: gaps[0], reason: "first eligible gap in document order" };
|
||||
}
|
||||
|
||||
// Cheap gate BEFORE burning test cycles. Reject if proposal smells wrong.
|
||||
// This catches runaway cloud proposals before we apply+test them.
|
||||
export function scoreProposal(p: Proposal): { accept: boolean; reason: string } {
|
||||
if (p.files.length === 0) return { accept: false, reason: "empty proposal" };
|
||||
if (p.files.length > 5) return { accept: false, reason: `>5 files (${p.files.length})` };
|
||||
if (p.estimated_loc > 200) return { accept: false, reason: `>200 LOC (${p.estimated_loc})` };
|
||||
|
||||
// Paths the bot is NEVER allowed to touch. Extend cautiously.
|
||||
const forbidden = [
|
||||
".git/", "secrets", "lakehouse.toml",
|
||||
"docs/ADR-", "docs/DECISIONS.md", "docs/PRD.md",
|
||||
"/etc/", "/root/", "Cargo.lock",
|
||||
];
|
||||
for (const f of p.files) {
|
||||
const hit = forbidden.find(prefix => f.path.includes(prefix));
|
||||
if (hit) return { accept: false, reason: `touches forbidden path (${hit}): ${f.path}` };
|
||||
}
|
||||
return { accept: true, reason: "passes size + path filters" };
|
||||
}
|
||||
|
||||
// Tests are green and changes applied. Do we actually open a PR?
|
||||
// Mem0-aligned: the cycle already short-circuited with `cycle_noop` if
|
||||
// every file was identical to current state, so by the time we're here
|
||||
// at least one file was actually added or updated.
|
||||
//
|
||||
// Stricter options below — uncomment to require a test file touched, etc.
|
||||
export function shouldOpenPR(c: CycleResult): { open: boolean; reason: string } {
|
||||
if (!c.testsGreen) return { open: false, reason: "tests red" };
|
||||
const realChanges = c.filesAdded.length + c.filesUpdated.length;
|
||||
if (realChanges === 0) return { open: false, reason: "nothing actually changed on disk" };
|
||||
|
||||
// Stricter option — require at least one test file in the diff:
|
||||
// const hasTest = [...c.filesAdded, ...c.filesUpdated].some(f => /test|_tests/i.test(f));
|
||||
// if (!hasTest) return { open: false, reason: "no test file touched" };
|
||||
|
||||
return { open: true, reason: `green + ${c.filesAdded.length} add / ${c.filesUpdated.length} update` };
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
// Gitea PR open. Reads PAT from ~/.git-credentials (set up by the
|
||||
// credential-helper flow). All PRs open as DRAFT so J reviews before
|
||||
// merge — never auto-merge.
|
||||
|
||||
import { readFile } from "node:fs/promises";
|
||||
|
||||
const GITEA_HOST = "https://git.agentview.dev";
|
||||
const REPO_OWNER = "profit";
|
||||
const REPO_NAME = "lakehouse";
|
||||
const CRED_FILE = "/home/profit/.git-credentials";
|
||||
|
||||
async function readPat(): Promise<string> {
|
||||
const raw = await readFile(CRED_FILE, "utf8");
|
||||
for (const line of raw.split("\n")) {
|
||||
const m = line.match(/^https:\/\/[^:]+:([^@]+)@git\.agentview\.dev/);
|
||||
if (m) return m[1];
|
||||
}
|
||||
throw new Error(`no Gitea PAT found in ${CRED_FILE}`);
|
||||
}
|
||||
|
||||
export interface OpenPrInput {
|
||||
branch: string; // head branch (just pushed)
|
||||
base?: string; // default "main"
|
||||
title: string;
|
||||
body: string;
|
||||
}
|
||||
|
||||
export interface OpenPrResult {
|
||||
number: number;
|
||||
html_url: string;
|
||||
}
|
||||
|
||||
export async function openPr(input: OpenPrInput): Promise<OpenPrResult> {
|
||||
const pat = await readPat();
|
||||
const url = `${GITEA_HOST}/api/v1/repos/${REPO_OWNER}/${REPO_NAME}/pulls`;
|
||||
const payload = {
|
||||
title: input.title,
|
||||
body: input.body,
|
||||
head: input.branch,
|
||||
base: input.base ?? "main",
|
||||
// Gitea uses "draft" flag — always on for bot PRs.
|
||||
draft: true,
|
||||
};
|
||||
const r = await fetch(url, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"content-type": "application/json",
|
||||
"Authorization": `token ${pat}`,
|
||||
},
|
||||
body: JSON.stringify(payload),
|
||||
signal: AbortSignal.timeout(30000),
|
||||
});
|
||||
if (!r.ok) {
|
||||
throw new Error(`Gitea ${r.status}: ${await r.text()}`);
|
||||
}
|
||||
const j = await r.json() as any;
|
||||
return { number: j.number, html_url: j.html_url };
|
||||
}
|
||||
+141
@@ -0,0 +1,141 @@
|
||||
// Gap detection + cloud proposal.
|
||||
//
|
||||
// Gap detection: scan docs/PRD.md for lines tagged [bot-eligible].
|
||||
// Each match becomes a Gap with surrounding context.
|
||||
//
|
||||
// Proposal: one-shot call to the T3 cloud model via the Python sidecar's
|
||||
// /generate endpoint. Asks for a structured JSON response with file
|
||||
// contents. Truncation-resistant via Phase 21's generate_continuable —
|
||||
// for now we pass max_tokens high and rely on the model completing in
|
||||
// one pass; swap to the Rust continuation wrapper if we see truncation.
|
||||
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { createHash } from "node:crypto";
|
||||
import type { Gap, Proposal } from "./types.ts";
|
||||
|
||||
const SIDECAR_URL = process.env.LH_SIDECAR_URL ?? "http://localhost:3200";
|
||||
const REPO_ROOT = "/home/profit/lakehouse";
|
||||
const PRD_PATH = `${REPO_ROOT}/docs/PRD.md`;
|
||||
const CLOUD_MODEL = process.env.LH_BOT_MODEL ?? "gpt-oss:120b";
|
||||
const MAX_TOKENS = 6000;
|
||||
|
||||
export async function findGaps(): Promise<Gap[]> {
|
||||
const prd = await readFile(PRD_PATH, "utf8");
|
||||
const lines = prd.split("\n");
|
||||
const gaps: Gap[] = [];
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
if (!lines[i].includes("[bot-eligible]")) continue;
|
||||
const contextLines = lines.slice(i, Math.min(i + 6, lines.length)).join("\n");
|
||||
const id = createHash("sha256").update(lines[i]).digest("hex").slice(0, 12);
|
||||
gaps.push({
|
||||
id,
|
||||
prd_line: lines[i].trim(),
|
||||
context: contextLines,
|
||||
source_file: "docs/PRD.md",
|
||||
line_number: i + 1,
|
||||
});
|
||||
}
|
||||
return gaps;
|
||||
}
|
||||
|
||||
const SYSTEM_PROMPT = `You are an assistant that proposes small, testable code changes to the Lakehouse repo.
|
||||
The Lakehouse is a Rust-first data platform with 13 crates + Bun/TypeScript test harness.
|
||||
You will be given one PRD gap tagged [bot-eligible] and must respond with a STRICT JSON object — no prose.
|
||||
|
||||
Rules:
|
||||
- Response MUST be a single JSON object, no markdown fences, no commentary.
|
||||
- Change MUST be small: <200 lines total, ≤5 files.
|
||||
- Include at least one test file (new or modified) that proves the change.
|
||||
- NEVER touch .git/, secrets, lakehouse.toml, docs/ADR-*, docs/DECISIONS.md, docs/PRD.md, /etc/, /root/, Cargo.lock.
|
||||
- Paths MUST be repo-relative (no leading /).
|
||||
- Whole-file contents only — no patches, no diffs.
|
||||
|
||||
Response shape:
|
||||
{
|
||||
"summary": "one line",
|
||||
"rationale": "why this addresses the gap",
|
||||
"files": [ { "path": "crates/foo/src/bar.rs", "content": "<full file>", "is_new": false } ],
|
||||
"estimated_loc": 42
|
||||
}`;
|
||||
|
||||
export async function generateProposal(gap: Gap, historySummary: string = ""): Promise<Proposal> {
|
||||
const sections = [
|
||||
`PRD gap (line ${gap.line_number}):`,
|
||||
"```",
|
||||
gap.context,
|
||||
"```",
|
||||
"",
|
||||
];
|
||||
if (historySummary) {
|
||||
sections.push(historySummary, "");
|
||||
}
|
||||
sections.push("Propose a small change that addresses this gap. Respond with the JSON object only.");
|
||||
const userPrompt = sections.join("\n");
|
||||
|
||||
const r = await fetch(`${SIDECAR_URL}/generate`, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
model: CLOUD_MODEL,
|
||||
system: SYSTEM_PROMPT,
|
||||
prompt: userPrompt,
|
||||
temperature: 0.2,
|
||||
max_tokens: MAX_TOKENS,
|
||||
think: false,
|
||||
}),
|
||||
signal: AbortSignal.timeout(180000), // cloud T3 can be slow — 3 min
|
||||
});
|
||||
if (!r.ok) {
|
||||
throw new Error(`sidecar ${r.status}: ${await r.text()}`);
|
||||
}
|
||||
const j = await r.json() as any;
|
||||
const raw: string = j.text ?? j.response ?? "";
|
||||
const usage = j.usage ?? {};
|
||||
const tokens = (usage.prompt_tokens ?? 0) + (usage.completion_tokens ?? 0);
|
||||
|
||||
const parsed = extractJson(raw);
|
||||
if (!parsed || typeof parsed !== "object") {
|
||||
throw new Error(`model returned no JSON object. Raw head: ${raw.slice(0, 300)}`);
|
||||
}
|
||||
if (!Array.isArray(parsed.files)) {
|
||||
throw new Error(`proposal.files not an array: ${JSON.stringify(parsed).slice(0, 200)}`);
|
||||
}
|
||||
|
||||
return {
|
||||
summary: String(parsed.summary ?? "").trim(),
|
||||
rationale: String(parsed.rationale ?? "").trim(),
|
||||
files: parsed.files.map((f: any) => ({
|
||||
path: String(f.path ?? ""),
|
||||
content: String(f.content ?? ""),
|
||||
is_new: Boolean(f.is_new),
|
||||
})).filter((f: any) => f.path && f.content !== undefined),
|
||||
estimated_loc: Number(parsed.estimated_loc ?? 0),
|
||||
model_used: CLOUD_MODEL,
|
||||
tokens_used: tokens,
|
||||
};
|
||||
}
|
||||
|
||||
// Find the first balanced JSON object in the string. Tolerates leading
|
||||
// "```json" fences even though we asked the model not to emit them.
|
||||
function extractJson(text: string): any | null {
|
||||
const cleaned = text.replace(/^```(?:json)?\s*/i, "").replace(/```\s*$/i, "").trim();
|
||||
let depth = 0;
|
||||
let start = -1;
|
||||
for (let i = 0; i < cleaned.length; i++) {
|
||||
const c = cleaned[i];
|
||||
if (c === "{") {
|
||||
if (depth === 0) start = i;
|
||||
depth++;
|
||||
} else if (c === "}") {
|
||||
depth--;
|
||||
if (depth === 0 && start >= 0) {
|
||||
try {
|
||||
return JSON.parse(cleaned.slice(start, i + 1));
|
||||
} catch {
|
||||
start = -1;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
+46
@@ -0,0 +1,46 @@
|
||||
// Run the test gate: `cargo test --workspace` + `bun test`.
|
||||
// Returns { green, output } — output is last ~4KB of combined stdout/stderr.
|
||||
|
||||
import { spawn } from "node:child_process";
|
||||
|
||||
const REPO_ROOT = "/home/profit/lakehouse";
|
||||
const TEST_TIMEOUT_MS = 15 * 60 * 1000; // 15 min cargo + 2 min bun, generous
|
||||
|
||||
export async function runTests(): Promise<{ green: boolean; output: string }> {
|
||||
const cargoOut = await runCmd("cargo", ["test", "--workspace", "--quiet", "--", "--test-threads=1"], REPO_ROOT, TEST_TIMEOUT_MS);
|
||||
if (cargoOut.code !== 0) {
|
||||
return { green: false, output: tail(`cargo test (${cargoOut.code}):\n${cargoOut.combined}`) };
|
||||
}
|
||||
|
||||
const bunOut = await runCmd("bun", ["test", "tests/multi-agent"], REPO_ROOT, 120000);
|
||||
const bunGreen = bunOut.code === 0;
|
||||
const combined = [
|
||||
`cargo test: OK`,
|
||||
`bun test (${bunOut.code}):`,
|
||||
bunOut.combined,
|
||||
].join("\n");
|
||||
return { green: bunGreen, output: tail(combined) };
|
||||
}
|
||||
|
||||
function tail(s: string, n = 4096): string {
|
||||
return s.length > n ? "…" + s.slice(-n) : s;
|
||||
}
|
||||
|
||||
function runCmd(
|
||||
cmd: string,
|
||||
args: string[],
|
||||
cwd: string,
|
||||
timeoutMs: number,
|
||||
): Promise<{ code: number; combined: string }> {
|
||||
return new Promise(resolveP => {
|
||||
const child = spawn(cmd, args, { cwd, env: { ...process.env } });
|
||||
let combined = "";
|
||||
child.stdout.on("data", d => { combined += d.toString(); });
|
||||
child.stderr.on("data", d => { combined += d.toString(); });
|
||||
const timer = setTimeout(() => child.kill("SIGKILL"), timeoutMs);
|
||||
child.on("close", code => {
|
||||
clearTimeout(timer);
|
||||
resolveP({ code: code ?? -1, combined });
|
||||
});
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
// Shared types for the PR-bot. Small and explicit — the bot's behavior
|
||||
// should be readable from these shapes alone.
|
||||
|
||||
export interface Gap {
|
||||
id: string; // hash of prd_line — stable key for dedup
|
||||
prd_line: string; // the [bot-eligible]-tagged line, verbatim
|
||||
context: string; // surrounding context from the PRD (next ~5 lines)
|
||||
source_file: string; // e.g. "docs/PRD.md"
|
||||
line_number: number;
|
||||
}
|
||||
|
||||
export interface ProposalFile {
|
||||
path: string; // repo-relative (no leading slash)
|
||||
content: string; // full file content — the bot writes whole files, not patches
|
||||
is_new: boolean; // true = create, false = overwrite existing
|
||||
}
|
||||
|
||||
export interface Proposal {
|
||||
summary: string; // one-line
|
||||
rationale: string; // why this change addresses the gap
|
||||
files: ProposalFile[];
|
||||
estimated_loc: number; // total added+changed lines across all files
|
||||
model_used: string;
|
||||
tokens_used: number;
|
||||
}
|
||||
|
||||
export interface CycleContext {
|
||||
startedAt: string; // ISO
|
||||
dailyCallsUsed: number;
|
||||
dailyCallsBudget: number;
|
||||
dailyTokensUsed: number;
|
||||
dailyTokensBudget: number;
|
||||
lastCycleAt: string | null;
|
||||
lastCycleGapId: string | null;
|
||||
autotuneBusy: boolean;
|
||||
workingTreeDirty: boolean;
|
||||
}
|
||||
|
||||
export type CycleOutcome =
|
||||
| "ok" // PR opened
|
||||
| "skipped_pause"
|
||||
| "skipped_cost"
|
||||
| "skipped_policy"
|
||||
| "skipped_no_gap"
|
||||
| "skipped_dirty_tree"
|
||||
| "proposal_rejected"
|
||||
| "apply_failed"
|
||||
| "cycle_noop" // proposal applied but every file was identical to what's on disk
|
||||
| "tests_failed"
|
||||
| "pr_skipped_by_policy"
|
||||
| "pr_failed"
|
||||
| "model_failed";
|
||||
|
||||
// Mem0-aligned apply outcomes. Three shapes instead of the binary
|
||||
// "written" / "errored". NOOP is a first-class outcome — identical
|
||||
// content shouldn't waste test cycles or open an empty PR.
|
||||
export type ApplyMode = "add" | "update" | "noop";
|
||||
|
||||
export interface ApplyOutcome {
|
||||
added: string[];
|
||||
updated: string[];
|
||||
noop: string[];
|
||||
errors: string[];
|
||||
}
|
||||
|
||||
export interface CycleResult {
|
||||
cycle_id: string;
|
||||
started_at: string;
|
||||
ended_at: string;
|
||||
outcome: CycleOutcome;
|
||||
reason: string;
|
||||
gap: Gap | null;
|
||||
proposal: Proposal | null;
|
||||
filesAdded: string[];
|
||||
filesUpdated: string[];
|
||||
filesNoop: string[];
|
||||
testsGreen: boolean | null; // null = not run
|
||||
testsOutput: string;
|
||||
prUrl: string | null;
|
||||
tokens_used: number;
|
||||
cloud_calls: number;
|
||||
}
|
||||
|
||||
export interface CostState {
|
||||
date: string; // YYYY-MM-DD UTC
|
||||
calls: number;
|
||||
tokens: number;
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"lockfileVersion": 1,
|
||||
"configVersion": 1,
|
||||
"workspaces": {
|
||||
"": {
|
||||
"dependencies": {
|
||||
"langfuse": "^3.38.20",
|
||||
},
|
||||
},
|
||||
},
|
||||
"packages": {
|
||||
"langfuse": ["langfuse@3.38.20", "", { "dependencies": { "langfuse-core": "^3.38.20" } }, "sha512-MAmBAASSzJtmK1O9HQegA1mFsQhT8Yf+OJRGvE7FXkyv3g/eiBE0glLD0Ohg3pkxhoPdggM5SejK7ue9ctlaMA=="],
|
||||
|
||||
"langfuse-core": ["langfuse-core@3.38.20", "", { "dependencies": { "mustache": "^4.2.0" } }, "sha512-zBKVmQN/1oT5VWZUBYlWzvokIlkC/6mnpgr/2atMyTeAm+jR3ia7w2iJMjlrF5/oG8ukO1s8+LDRCzJpF1QeEA=="],
|
||||
|
||||
"mustache": ["mustache@4.2.0", "", { "bin": { "mustache": "bin/mustache" } }, "sha512-71ippSywq5Yb7/tVYyGbkBggbU8H3u5Rz56fH60jGFgr8uHwxs+aSKeqmluIVzM0m0kB7xQjKS6qPfd0b2ZoqQ=="],
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,193 @@
|
||||
{
|
||||
"_description": "Lakehouse model matrix — authoritative routing for all agent tiers. Local models do the heavy lifting; cloud models are consulted sparingly for overview, strategic, and gatekeeper decisions. Read by tests/multi-agent/scenario.ts and mcp-server/index.ts.",
|
||||
"version": 1,
|
||||
"updated": "2026-04-21",
|
||||
"providers": {
|
||||
"ollama_local": {
|
||||
"base_url": "http://localhost:11434",
|
||||
"key_env": null
|
||||
},
|
||||
"ollama_cloud": {
|
||||
"base_url": "https://ollama.com",
|
||||
"key_env": "OLLAMA_CLOUD_KEY",
|
||||
"key_source": "/root/llm_team_config.json → providers.ollama_cloud.api_key",
|
||||
"rate_budget": {
|
||||
"calls_per_hour": 200,
|
||||
"calls_per_day": 2000,
|
||||
"notes": "Paid tier — generous. Policy: keep overview calls ≤ 3/scenario, strategic ≤ 10/day, gatekeeper ≤ 5/day."
|
||||
}
|
||||
}
|
||||
},
|
||||
"tiers": {
|
||||
"t1_hot": {
|
||||
"purpose": "Per tool call — SQL generation, hybrid_search, sql(). Runs 50-200 times per scenario. Latency-sensitive.",
|
||||
"kind": "local_fast",
|
||||
"primary": { "model": "qwen3.5:latest", "provider": "ollama_local", "context_window": 262144 },
|
||||
"fallback": { "model": "qwen2.5:latest", "provider": "ollama_local", "context_window": 32768 },
|
||||
"max_tokens": 1000,
|
||||
"temperature": 0.3,
|
||||
"never_route_cloud": true,
|
||||
"context_budget": {
|
||||
"system_prompt_cap": 4000,
|
||||
"prior_context_cap": 6000,
|
||||
"tool_results_cap": 8000,
|
||||
"safety_margin": 2000,
|
||||
"overflow_policy": "summarize_oldest_tool_results_via_t3"
|
||||
},
|
||||
"rationale": "qwen3.5:latest is a 9.7B thinking model with 262K context that emits clean JSON (verified 2026-04-21 after mistral A/B wipeout). Thinking budget requires max_tokens ≥ 400; we hold 1000 to keep propose_done payloads intact. qwen2.5 stays as fallback."
|
||||
},
|
||||
"t2_review": {
|
||||
"purpose": "Per step consensus — executor ↔ reviewer loop critique. 5-14 calls per event.",
|
||||
"kind": "local_balanced",
|
||||
"primary": { "model": "qwen3:latest", "provider": "ollama_local", "context_window": 40960 },
|
||||
"fallback": { "model": "qwen2.5:latest", "provider": "ollama_local", "context_window": 32768 },
|
||||
"max_tokens": 800,
|
||||
"temperature": 0.3,
|
||||
"never_route_cloud": true,
|
||||
"context_budget": {
|
||||
"system_prompt_cap": 2000,
|
||||
"recent_turns_cap": 4000,
|
||||
"safety_margin": 1000
|
||||
},
|
||||
"rationale": "Reviewer only needs to detect schema violations and drift — a 7B model is sufficient."
|
||||
},
|
||||
"t3_overview": {
|
||||
"purpose": "Mid-day checkpoint after every misplacement + every Nth event, and cross-day lesson. 1-3 calls per scenario.",
|
||||
"kind": "thinking_cloud",
|
||||
"primary": { "model": "gpt-oss:120b", "provider": "ollama_cloud", "context_window": 131072 },
|
||||
"local_fallback": { "model": "gpt-oss:20b", "provider": "ollama_local", "context_window": 131072 },
|
||||
"max_tokens": 900,
|
||||
"temperature": 0.2,
|
||||
"cloud_budget_per_scenario": 5,
|
||||
"env_flag": "LH_OVERVIEW_CLOUD=1",
|
||||
"context_budget": {
|
||||
"event_digest_cap": 30000,
|
||||
"checkpoint_cap": 8000,
|
||||
"lesson_corpus_cap": 40000,
|
||||
"safety_margin": 8000,
|
||||
"overflow_policy": "chunk_lessons_via_cosine_topk"
|
||||
},
|
||||
"rationale": "Same prompt family as local 20b (gpt-oss series) — prompts port directly. 120b is faster via cloud than 20b local in practice, and lessons are noticeably more specific."
|
||||
},
|
||||
"t4_strategic": {
|
||||
"purpose": "Daily playbook board re-ranking, weekly gap audit, pattern discovery across accumulated playbooks. 1-10 calls per day.",
|
||||
"kind": "thinking_cloud_large",
|
||||
"primary": { "model": "kimi-k2.6", "provider": "ollama_cloud", "context_window": 200000 },
|
||||
"secondary": { "model": "qwen3.5:397b", "provider": "ollama_cloud", "context_window": 131072 },
|
||||
"fallback": { "model": "glm-4.7", "provider": "ollama_cloud", "context_window": 131072 },
|
||||
"local_fallback": { "model": "gpt-oss:20b", "provider": "ollama_local", "context_window": 131072 },
|
||||
"max_tokens": 2000,
|
||||
"temperature": 0.2,
|
||||
"cloud_budget_per_day": 10,
|
||||
"context_budget": {
|
||||
"playbook_corpus_cap": 80000,
|
||||
"pattern_history_cap": 20000,
|
||||
"safety_margin": 16000,
|
||||
"overflow_policy": "two_pass_map_reduce"
|
||||
},
|
||||
"rationale": "J named qwen3.5 specifically. GLM-4.7 is a promising alternate for debate phase. Runs after all scenarios complete for the day."
|
||||
},
|
||||
"t5_gatekeeper": {
|
||||
"purpose": "MUST route here: architecture changes, new client onboarding, schema migrations, playbook retirements, index rebuilds, autotune config changes.",
|
||||
"kind": "thinking_cloud_deepest",
|
||||
"primary": { "model": "kimi-k2-thinking", "provider": "ollama_cloud", "context_window": 200000 },
|
||||
"fallback": { "model": "deepseek-v3.1:671b", "provider": "ollama_cloud", "context_window": 131072 },
|
||||
"secondary_fallback": { "model": "qwen3.5:397b", "provider": "ollama_cloud", "context_window": 131072 },
|
||||
"local_fallback": { "model": "gpt-oss:20b", "provider": "ollama_local", "context_window": 131072 },
|
||||
"max_tokens": 4000,
|
||||
"temperature": 0.1,
|
||||
"cloud_budget_per_day": 5,
|
||||
"audit_log": true,
|
||||
"context_budget": {
|
||||
"decision_doc_cap": 50000,
|
||||
"evidence_bundle_cap": 100000,
|
||||
"prior_gatekeeper_decisions_cap": 20000,
|
||||
"safety_margin": 20000,
|
||||
"overflow_policy": "escalate_to_kimi_k2_1t_or_split_decision"
|
||||
},
|
||||
"rationale": "Highest-stakes decisions — reasoning depth matters more than latency. Audit log so J can always see what the gatekeeper was asked and what it answered. No human approval required today; escalate later if mis-decisions show up."
|
||||
}
|
||||
},
|
||||
|
||||
"embeddings": {
|
||||
"_description": "Vector embedding models for hybrid search + playbook_memory similarity. Always local — embedding calls are too high-volume for cloud. qwen3-embedding is the new primary (2026-04-21); nomic-embed-text-v2-moe is the MoE alternative for corpus-scale embedding; original nomic-embed-text stays as legacy fallback while vectord indexes are being re-embedded.",
|
||||
"primary": { "model": "qwen3-embedding", "provider": "ollama_local", "dim": 1024, "context_window": 32768 },
|
||||
"alternate": { "model": "nomic-embed-text-v2-moe", "provider": "ollama_local", "dim": 768, "context_window": 2048, "notes": "MoE variant — better for long documents, slower per-call, roughly matches nomic-embed-text on short text." },
|
||||
"legacy": { "model": "nomic-embed-text:latest", "provider": "ollama_local", "dim": 768, "context_window": 8192, "notes": "Used by existing vectord indexes. Do NOT switch live until indexes are rebuilt, or hybrid search recall will crater." },
|
||||
"migration_plan": "Re-embed workers_500k (and all production indexes) with qwen3-embedding, keep the old index under a _v1 suffix for rollback, flip the profile pointer only after recall benchmarks match or exceed legacy. Touches vectord::agent + autotune."
|
||||
},
|
||||
|
||||
"context_management": {
|
||||
"_description": "Rule zero: NEVER call a model with more tokens than its context_window minus safety_margin. Every call goes through the budget checker first. If over budget → chunk, summarize, or escalate. This is the stability floor.",
|
||||
"token_estimator": {
|
||||
"method": "chars_div_4",
|
||||
"note": "Rough, biased safe by ~15%. For production, swap to tiktoken or the provider's tokenizer endpoint."
|
||||
},
|
||||
"overflow_policies": {
|
||||
"summarize_oldest_tool_results_via_t3": {
|
||||
"when": "T1 conversation history + tool results exceed context_budget.tool_results_cap",
|
||||
"how": "Send oldest N tool results to T3 with prompt 'summarize these into 500 tokens that preserve what the executor needs to know'; replace them with the summary in the running conversation."
|
||||
},
|
||||
"chunk_lessons_via_cosine_topk": {
|
||||
"when": "lesson corpus in data/_playbook_lessons/*.json exceeds lesson_corpus_cap",
|
||||
"how": "Embed the current scenario spec, cosine-rank all lessons, take top-K until budget exhausted. Fall back to date-sorted if embeddings unavailable."
|
||||
},
|
||||
"two_pass_map_reduce": {
|
||||
"when": "T4 playbook corpus exceeds playbook_corpus_cap",
|
||||
"how": "Pass 1: chunk playbooks into ≤30K token shards, run primary model on each shard to emit 'shard summary'. Pass 2: feed all summaries to primary model for global synthesis. Logged as two audit entries."
|
||||
},
|
||||
"escalate_to_kimi_k2_1t_or_split_decision": {
|
||||
"when": "T5 decision evidence exceeds decision_doc_cap + evidence_bundle_cap",
|
||||
"how": "Prefer kimi-k2:1t which has 1M context. If still over, split decision into sub-decisions (e.g. 'retire playbooks by city' instead of 'retire playbooks globally') and loop."
|
||||
}
|
||||
},
|
||||
"chunking_cache": {
|
||||
"_description": "Precomputed shards of the playbook corpus, indexed by (corpus_version, shard_id). Avoids re-chunking on every T4 run.",
|
||||
"location": "data/_chunk_cache/",
|
||||
"invalidation": "Key includes corpus hash. When playbook_memory changes, the hash changes, the cache misses, and chunks regenerate.",
|
||||
"implementation_status": "SPEC — next sprint."
|
||||
},
|
||||
"implementation_status": "context_window + context_budget fields WIRED in config. Enforcement helper NOT yet wired in agent.ts. Next: add estimateTokens() + budgetCheck() helpers, route all generate() calls through them, emit a metric when overflow policy fires."
|
||||
},
|
||||
"experimental_rotation": {
|
||||
"enabled": false,
|
||||
"purpose": "Sample newer models on a schedule to collect comparison data without rate-limit risk.",
|
||||
"candidates": [
|
||||
{ "model": "minimax-m2.7", "notes": "Newer minimax; unknown output stability" },
|
||||
{ "model": "glm-5", "notes": "GLM next-gen; larger context" },
|
||||
{ "model": "glm-5.1", "notes": "Incremental on GLM-5" },
|
||||
{ "model": "qwen3-next:80b", "notes": "Qwen's experimental successor; smaller than 3.5" },
|
||||
{ "model": "qwen3-coder-next", "notes": "Coder-optimized — good for SQL gen T1 experiments" },
|
||||
{ "model": "deepseek-v3.2", "notes": "Smaller deepseek; reasoning/coding" },
|
||||
{ "model": "nemotron-3-super", "notes": "NVIDIA 230B; general" },
|
||||
{ "model": "cogito-2.1:671b", "notes": "671B general" },
|
||||
{ "model": "mistral-large-3:675b", "notes": "Mistral's flagship; good T3 candidate" }
|
||||
],
|
||||
"rotation": "weekly",
|
||||
"sample_rate": 0.1,
|
||||
"apply_to_tier": "t4_strategic",
|
||||
"notes": "When enabled, T4 routes 10% of calls to a rotating experimental model. Log comparison in /data/_model_eval/ — if the experimental consistently beats primary across 3 rotations, promote it to primary."
|
||||
},
|
||||
"playbook_versioning": {
|
||||
"enabled": true,
|
||||
"purpose": "A playbook can work, then break when architecture changes. Versioning lets us pin which change retired which playbook.",
|
||||
"dataset": "playbook_memory",
|
||||
"schema_additions": {
|
||||
"version": "integer — auto-increment per operation",
|
||||
"parent_id": "string — previous version entry_id for same operation (null for v1)",
|
||||
"retired_at": "timestamp — set when success_rate drops or architecture changes",
|
||||
"retirement_reason": "string — e.g. 'schema_migration:workers_500k 2026-05-03'",
|
||||
"architecture_snapshot": "object — crate versions, index name, schema fingerprint at seed time"
|
||||
},
|
||||
"retire_triggers": [
|
||||
"success_rate < 0.3 over last 20 citations",
|
||||
"schema_fingerprint mismatch detected at retrieval time",
|
||||
"architecture change event emitted by ingestd/vectord",
|
||||
"T5 gatekeeper explicitly retires via /vectors/playbook_memory/retire"
|
||||
],
|
||||
"read_back_policy": "retrieval returns only non-retired versions. History endpoint /vectors/playbook_memory/history/{operation} returns the full chain.",
|
||||
"ui_surface": "mcp-server to render a diff view: side-by-side of playbook versions with a timeline of what changed and when.",
|
||||
"implementation_status": "SPEC — not yet wired. Target: next sprint. Touches gateway + catalogd + mcp-server."
|
||||
},
|
||||
"matrix_index_hybrid_search_note": "Phase 22 candidate: elevate the hybrid_search T1 tool to consult T3 when a pool returns <3 matches OR when the same (role, city) combo has failed N times in 24h. Consult result is a reformulated sql_filter the executor retries with. Keeps T1 fast on the happy path, escalates to T3 only on low-recall signals."
|
||||
}
|
||||
@@ -25,7 +25,7 @@ pub struct EmbedResponse {
|
||||
pub dimensions: usize,
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize)]
|
||||
#[derive(Clone, Serialize, Deserialize)]
|
||||
pub struct GenerateRequest {
|
||||
pub prompt: String,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
@@ -36,6 +36,14 @@ pub struct GenerateRequest {
|
||||
pub temperature: Option<f64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub max_tokens: Option<u32>,
|
||||
/// Phase 21 — per-call opt-out of hidden reasoning. Thinking models
|
||||
/// (qwen3.5, gpt-oss, etc) burn tokens on reasoning before the
|
||||
/// visible response starts; setting this to `false` on hot-path
|
||||
/// JSON emitters avoids empty returns when the budget is tight.
|
||||
/// Sidecar forwards this to Ollama's `think` parameter; if the
|
||||
/// sidecar drops an unknown field the request still succeeds.
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub think: Option<bool>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize, Serialize, Clone)]
|
||||
|
||||
@@ -0,0 +1,194 @@
|
||||
//! Phase 21 — context-budget accounting for model calls.
|
||||
//!
|
||||
//! Ports `assertContextBudget` + `estimateTokens` + `CONTEXT_WINDOWS`
|
||||
//! from `tests/multi-agent/agent.ts` so Rust-side callers (gateway
|
||||
//! tool surfaces, future Rust agents) get the same loud-fail behavior
|
||||
//! on window overflow instead of silent truncation.
|
||||
//!
|
||||
//! The token estimator is deliberately the same chars/4 heuristic as
|
||||
//! the TS side. It's biased ~15% safe — pessimistic on English, correct
|
||||
//! within a factor of 2 on code. Swap to a provider tokenizer only when
|
||||
//! the estimator drives a decision (we're nowhere near that yet).
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::OnceLock;
|
||||
|
||||
/// Rough token count. `chars / 4` ceiling. See module docs for why
|
||||
/// this heuristic is sufficient.
|
||||
pub fn estimate_tokens(text: &str) -> usize {
|
||||
(text.chars().count() + 3) / 4
|
||||
}
|
||||
|
||||
/// Phase 21 — per-model context windows, mirroring the TS table in
|
||||
/// `tests/multi-agent/agent.ts`. Anchored on each model's documented
|
||||
/// max; unknown models fall back to `DEFAULT_CONTEXT_WINDOW`.
|
||||
pub const DEFAULT_CONTEXT_WINDOW: usize = 32_768;
|
||||
pub const DEFAULT_SAFETY_MARGIN: usize = 2_000;
|
||||
pub const DEFAULT_MAX_TOKENS: usize = 800;
|
||||
|
||||
fn known_windows() -> &'static HashMap<&'static str, usize> {
|
||||
static TABLE: OnceLock<HashMap<&'static str, usize>> = OnceLock::new();
|
||||
TABLE.get_or_init(|| {
|
||||
let mut m = HashMap::new();
|
||||
m.insert("mistral:latest", 32_768);
|
||||
m.insert("qwen2.5:latest", 32_768);
|
||||
m.insert("qwen3:latest", 40_960);
|
||||
m.insert("qwen3.5:latest", 262_144);
|
||||
m.insert("qwen3-embedding", 32_768);
|
||||
m.insert("nomic-embed-text-v2-moe", 2_048);
|
||||
m.insert("gpt-oss:20b", 131_072);
|
||||
m.insert("gpt-oss:120b", 131_072);
|
||||
m.insert("qwen3.5:397b", 131_072);
|
||||
m.insert("kimi-k2-thinking", 200_000);
|
||||
m.insert("kimi-k2.6", 200_000);
|
||||
m.insert("kimi-k2:1t", 1_048_576);
|
||||
m.insert("deepseek-v3.1:671b", 131_072);
|
||||
m.insert("glm-4.7", 131_072);
|
||||
m
|
||||
})
|
||||
}
|
||||
|
||||
pub fn context_window_for(model: &str) -> usize {
|
||||
known_windows().get(model).copied().unwrap_or(DEFAULT_CONTEXT_WINDOW)
|
||||
}
|
||||
|
||||
/// Result of a budget check — exposes the numbers so callers can log
|
||||
/// how much headroom remains without re-running the estimator.
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub struct BudgetCheck {
|
||||
pub estimated: usize,
|
||||
pub window: usize,
|
||||
pub remaining: i64,
|
||||
}
|
||||
|
||||
/// Inputs to `assert_context_budget`. `bypass` exists for call sites
|
||||
/// that handle their own overflow (continuation's second pass already
|
||||
/// counted the partial; T5 gatekeeper prompts have a separate policy).
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct BudgetOpts<'a> {
|
||||
pub system: Option<&'a str>,
|
||||
pub max_tokens: Option<usize>,
|
||||
pub safety_margin: Option<usize>,
|
||||
pub bypass: bool,
|
||||
}
|
||||
|
||||
/// Phase 21's loud-fail primitive. Returns a `BudgetCheck` on success
|
||||
/// and the same struct plus over-by count on failure. The whole point
|
||||
/// is to stop silent truncation — callers that expect overflow should
|
||||
/// chunk BEFORE calling or set `bypass: true`.
|
||||
pub fn assert_context_budget(
|
||||
model: &str,
|
||||
prompt: &str,
|
||||
opts: BudgetOpts,
|
||||
) -> Result<BudgetCheck, (BudgetCheck, usize)> {
|
||||
let window = context_window_for(model);
|
||||
let safety = opts.safety_margin.unwrap_or(DEFAULT_SAFETY_MARGIN);
|
||||
let max_tokens = opts.max_tokens.unwrap_or(DEFAULT_MAX_TOKENS);
|
||||
let sys_tokens = opts.system.map(estimate_tokens).unwrap_or(0);
|
||||
let estimated = estimate_tokens(prompt) + sys_tokens + max_tokens;
|
||||
let remaining = window as i64 - estimated as i64 - safety as i64;
|
||||
let check = BudgetCheck { estimated, window, remaining };
|
||||
if remaining < 0 && !opts.bypass {
|
||||
return Err((check, (-remaining) as usize));
|
||||
}
|
||||
Ok(check)
|
||||
}
|
||||
|
||||
/// Convenience — format an overflow error the same way the TS side
|
||||
/// does. Exposed so downstream crates render consistent messages.
|
||||
pub fn overflow_message(model: &str, check: &BudgetCheck, over_by: usize, safety: usize) -> String {
|
||||
format!(
|
||||
"context overflow: model={} est={}t window={}t safety={}t over={}t. \
|
||||
Chunk the prompt (see config/models.json overflow_policies) or set \
|
||||
bypass:true if you know the risk.",
|
||||
model, check.estimated, check.window, safety, over_by,
|
||||
)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn estimate_tokens_ceiling_divides_by_four() {
|
||||
assert_eq!(estimate_tokens(""), 0);
|
||||
assert_eq!(estimate_tokens("abc"), 1); // 3 → ceil(3/4) = 1
|
||||
assert_eq!(estimate_tokens("abcd"), 1); // 4 → ceil(4/4) = 1
|
||||
assert_eq!(estimate_tokens("abcde"), 2); // 5 → ceil(5/4) = 2
|
||||
assert_eq!(estimate_tokens(&"x".repeat(400)), 100);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn context_window_known_and_fallback() {
|
||||
assert_eq!(context_window_for("qwen3.5:latest"), 262_144);
|
||||
assert_eq!(context_window_for("kimi-k2:1t"), 1_048_576);
|
||||
assert_eq!(context_window_for("some-unreleased-model"), DEFAULT_CONTEXT_WINDOW);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn budget_passes_well_under_window() {
|
||||
let check = assert_context_budget(
|
||||
"qwen3:latest",
|
||||
&"x".repeat(4_000), // ~1000 tokens
|
||||
BudgetOpts { max_tokens: Some(500), ..Default::default() },
|
||||
).expect("well under 40K window");
|
||||
assert!(check.remaining > 30_000);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn budget_fails_when_prompt_overflows_window() {
|
||||
let huge = "x".repeat(200_000); // ~50K tokens, over qwen3's 40K
|
||||
let err = assert_context_budget(
|
||||
"qwen3:latest",
|
||||
&huge,
|
||||
BudgetOpts::default(),
|
||||
).expect_err("should overflow qwen3's 40K window");
|
||||
assert!(err.1 > 0, "over_by must be positive");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn budget_bypass_returns_ok_even_over() {
|
||||
let huge = "x".repeat(200_000);
|
||||
let check = assert_context_budget(
|
||||
"qwen3:latest",
|
||||
&huge,
|
||||
BudgetOpts { bypass: true, ..Default::default() },
|
||||
).expect("bypass must suppress the error");
|
||||
assert!(check.remaining < 0, "check still reports negative remaining");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn budget_counts_system_prompt() {
|
||||
// 10K-char system prompt → ~2500 tokens. With a big max_tokens
|
||||
// this should push us closer to the window.
|
||||
let sys = "s".repeat(10_000);
|
||||
let prompt = "p".repeat(4_000);
|
||||
let with_sys = assert_context_budget(
|
||||
"qwen3:latest",
|
||||
&prompt,
|
||||
BudgetOpts {
|
||||
system: Some(&sys),
|
||||
max_tokens: Some(500),
|
||||
..Default::default()
|
||||
},
|
||||
).unwrap();
|
||||
let without_sys = assert_context_budget(
|
||||
"qwen3:latest",
|
||||
&prompt,
|
||||
BudgetOpts { max_tokens: Some(500), ..Default::default() },
|
||||
).unwrap();
|
||||
assert!(with_sys.estimated > without_sys.estimated,
|
||||
"system prompt should raise estimate");
|
||||
assert_eq!(with_sys.estimated - without_sys.estimated, estimate_tokens(&sys));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn overflow_message_includes_numbers() {
|
||||
let check = BudgetCheck { estimated: 42_000, window: 40_960, remaining: -1_040 };
|
||||
let msg = overflow_message("qwen3:latest", &check, 3_040, 2_000);
|
||||
assert!(msg.contains("qwen3:latest"));
|
||||
assert!(msg.contains("42000t"));
|
||||
assert!(msg.contains("40960t"));
|
||||
assert!(msg.contains("3040t"));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,438 @@
|
||||
//! Phase 21 — OUTPUT-overflow handler. Ports `generateContinuable`
|
||||
//! from `tests/multi-agent/agent.ts`.
|
||||
//!
|
||||
//! Two failure modes to repair:
|
||||
//!
|
||||
//! * EMPTY response — thinking model ate the entire budget on hidden
|
||||
//! reasoning before emitting a token. Fix: retry the original prompt
|
||||
//! with 2× the budget, geometric up to `BUDGET_CAP`.
|
||||
//!
|
||||
//! * TRUNCATED non-empty — model got most of the way but hit
|
||||
//! max_tokens before closing the structure. Fix: continue with the
|
||||
//! partial response in the prompt as scratchpad, so the model knows
|
||||
//! where to pick up without restarting.
|
||||
//!
|
||||
//! `TextGenerator` abstracts the sidecar so tests can inject canned
|
||||
//! responses without a live Ollama.
|
||||
|
||||
use std::future::Future;
|
||||
|
||||
use crate::client::{AiClient, GenerateRequest, GenerateResponse};
|
||||
|
||||
/// Shape classifier for `is_structurally_complete`. JSON responses
|
||||
/// must parse; text responses just need to be non-empty.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum ResponseShape {
|
||||
Json,
|
||||
Text,
|
||||
}
|
||||
|
||||
/// Trait that `generate_continuable` + `generate_tree_split` call. The
|
||||
/// real implementation forwards to `AiClient::generate`; tests supply a
|
||||
/// mock with a scripted sequence of responses.
|
||||
pub trait TextGenerator: Send + Sync {
|
||||
fn generate_text(
|
||||
&self,
|
||||
req: GenerateRequest,
|
||||
) -> impl Future<Output = Result<GenerateResponse, String>> + Send;
|
||||
}
|
||||
|
||||
impl TextGenerator for AiClient {
|
||||
fn generate_text(
|
||||
&self,
|
||||
req: GenerateRequest,
|
||||
) -> impl Future<Output = Result<GenerateResponse, String>> + Send {
|
||||
self.generate(req)
|
||||
}
|
||||
}
|
||||
|
||||
/// Strip a surrounding ```json``` fence if present. Leaves the inner
|
||||
/// content alone otherwise. Returns a slice of `s`.
|
||||
fn strip_json_fence(s: &str) -> &str {
|
||||
let t = s.trim();
|
||||
if let Some(rest) = t.strip_prefix("```json") {
|
||||
rest.trim_start_matches('\n').strip_suffix("```").unwrap_or(rest).trim()
|
||||
} else if let Some(rest) = t.strip_prefix("```") {
|
||||
rest.trim_start_matches('\n').strip_suffix("```").unwrap_or(rest).trim()
|
||||
} else {
|
||||
t
|
||||
}
|
||||
}
|
||||
|
||||
/// Port of the TS brace-balance + JSON.parse check. Returns true when
|
||||
/// the outermost `{...}` block is balanced and parses. Text shape is
|
||||
/// satisfied by any non-empty, non-whitespace payload.
|
||||
pub fn is_structurally_complete(text: &str, shape: ResponseShape) -> bool {
|
||||
if text.trim().is_empty() { return false; }
|
||||
if shape == ResponseShape::Text { return true; }
|
||||
|
||||
let s = strip_json_fence(text);
|
||||
let Some(start) = s.find('{') else { return false };
|
||||
let Some(end) = s.rfind('}') else { return false };
|
||||
if end <= start { return false; }
|
||||
let slice = &s[start..=end];
|
||||
|
||||
// Balance check — cheaper than parse, catches truncated nests.
|
||||
// String state tracked because `{` inside a string doesn't count.
|
||||
let mut depth: i32 = 0;
|
||||
let mut in_str = false;
|
||||
let mut esc = false;
|
||||
for c in slice.chars() {
|
||||
if esc { esc = false; continue; }
|
||||
if c == '\\' { esc = true; continue; }
|
||||
if c == '"' { in_str = !in_str; continue; }
|
||||
if in_str { continue; }
|
||||
if c == '{' { depth += 1; }
|
||||
else if c == '}' {
|
||||
depth -= 1;
|
||||
if depth < 0 { return false; }
|
||||
}
|
||||
}
|
||||
if depth != 0 { return false; }
|
||||
// Parse check is the tie-breaker — balanced but invalid JSON (e.g.
|
||||
// trailing comma before `}`) shouldn't count as complete.
|
||||
serde_json::from_str::<serde_json::Value>(slice).is_ok()
|
||||
}
|
||||
|
||||
/// Knobs for `generate_continuable`. All optional with sensible
|
||||
/// defaults that match the TS version.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct ContinuableOpts {
|
||||
pub model: String,
|
||||
pub max_tokens: Option<u32>,
|
||||
pub temperature: Option<f64>,
|
||||
pub system: Option<String>,
|
||||
pub shape: ResponseShape,
|
||||
pub max_continuations: usize,
|
||||
pub think: Option<bool>,
|
||||
/// Geometric-backoff ceiling for the empty-response retry path.
|
||||
/// Matches TS's `budgetCap = 8000`.
|
||||
pub budget_cap: u32,
|
||||
/// Maximum empty-response retries before giving up. Matches TS's
|
||||
/// hardcoded `retry < 3`.
|
||||
pub max_empty_retries: usize,
|
||||
}
|
||||
|
||||
impl ContinuableOpts {
|
||||
pub fn new(model: impl Into<String>) -> Self {
|
||||
Self {
|
||||
model: model.into(),
|
||||
max_tokens: None,
|
||||
temperature: None,
|
||||
system: None,
|
||||
shape: ResponseShape::Json,
|
||||
max_continuations: 3,
|
||||
think: None,
|
||||
budget_cap: 8_000,
|
||||
max_empty_retries: 3,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Outcome of a `generate_continuable` call. Carries the combined
|
||||
/// text plus diagnostic counters so observability downstream can
|
||||
/// report "how many continuations did that query cost".
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct ContinuableOutcome {
|
||||
pub text: String,
|
||||
pub empty_retries: usize,
|
||||
pub continuations: usize,
|
||||
pub final_complete: bool,
|
||||
}
|
||||
|
||||
fn make_request(opts: &ContinuableOpts, prompt: String, current_max: u32) -> GenerateRequest {
|
||||
GenerateRequest {
|
||||
prompt,
|
||||
model: Some(opts.model.clone()),
|
||||
system: opts.system.clone(),
|
||||
temperature: opts.temperature,
|
||||
max_tokens: Some(current_max),
|
||||
think: opts.think,
|
||||
}
|
||||
}
|
||||
|
||||
fn continuation_prompt(original: &str, partial: &str) -> String {
|
||||
format!(
|
||||
"{original}\n\n\
|
||||
PARTIAL RESPONSE SO FAR (continue from here — do NOT restart, \
|
||||
do NOT repeat what's already there, emit ONLY the remaining \
|
||||
tokens to close the structure):\n{partial}"
|
||||
)
|
||||
}
|
||||
|
||||
/// Phase 21 — output-overflow safe generate. See module docs for the
|
||||
/// two failure modes repaired. On final-still-incomplete, returns the
|
||||
/// combined text with `final_complete: false` so the caller's parser
|
||||
/// can throw with the raw text for forensics rather than silently
|
||||
/// truncating.
|
||||
pub async fn generate_continuable<G: TextGenerator>(
|
||||
generator: &G,
|
||||
prompt: &str,
|
||||
opts: &ContinuableOpts,
|
||||
) -> Result<ContinuableOutcome, String> {
|
||||
let initial_max = opts.max_tokens.unwrap_or(800);
|
||||
let mut current_max = initial_max;
|
||||
let mut combined = String::new();
|
||||
let mut empty_retries = 0usize;
|
||||
let mut continuations = 0usize;
|
||||
|
||||
// Phase 21(a) — empty-response backoff loop.
|
||||
for retry in 0..opts.max_empty_retries {
|
||||
let req = make_request(opts, prompt.to_string(), current_max);
|
||||
let resp = generator.generate_text(req).await?;
|
||||
if !resp.text.trim().is_empty() {
|
||||
combined = resp.text;
|
||||
break;
|
||||
}
|
||||
empty_retries = retry + 1;
|
||||
current_max = (current_max.saturating_mul(2)).min(opts.budget_cap);
|
||||
}
|
||||
|
||||
// Phase 21(b) — structural-completion continuation loop. Runs on
|
||||
// the truncated-non-empty case; empty + exhausted retries falls
|
||||
// through with empty combined and final_complete=false.
|
||||
for _ in 0..opts.max_continuations {
|
||||
if is_structurally_complete(&combined, opts.shape) {
|
||||
return Ok(ContinuableOutcome {
|
||||
text: combined,
|
||||
empty_retries,
|
||||
continuations,
|
||||
final_complete: true,
|
||||
});
|
||||
}
|
||||
if combined.trim().is_empty() {
|
||||
// Nothing to continue from — continuing "" is identical to
|
||||
// the initial call and would loop. Bail so the caller sees
|
||||
// the failure rather than burning N extra calls.
|
||||
break;
|
||||
}
|
||||
let cont_prompt = continuation_prompt(prompt, &combined);
|
||||
let req = make_request(opts, cont_prompt, current_max.min(opts.budget_cap));
|
||||
let resp = generator.generate_text(req).await?;
|
||||
combined.push_str(&resp.text);
|
||||
continuations += 1;
|
||||
}
|
||||
|
||||
let final_complete = is_structurally_complete(&combined, opts.shape);
|
||||
Ok(ContinuableOutcome {
|
||||
text: combined,
|
||||
empty_retries,
|
||||
continuations,
|
||||
final_complete,
|
||||
})
|
||||
}
|
||||
|
||||
/// Scripted generator for unit tests. Returns responses from `script`
|
||||
/// in order; extra calls reuse the last entry so tests don't have to
|
||||
/// count past what they actually assert on.
|
||||
#[cfg(test)]
|
||||
pub struct ScriptedGenerator {
|
||||
script: Vec<Result<String, String>>,
|
||||
calls: std::sync::Arc<std::sync::Mutex<Vec<GenerateRequest>>>,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
impl ScriptedGenerator {
|
||||
pub fn new<I>(script: I) -> Self
|
||||
where
|
||||
I: IntoIterator<Item = Result<String, String>>,
|
||||
{
|
||||
Self {
|
||||
script: script.into_iter().collect(),
|
||||
calls: std::sync::Arc::new(std::sync::Mutex::new(Vec::new())),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn calls(&self) -> Vec<GenerateRequest> {
|
||||
self.calls.lock().unwrap().clone()
|
||||
}
|
||||
|
||||
pub fn call_count(&self) -> usize {
|
||||
self.calls.lock().unwrap().len()
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
impl TextGenerator for ScriptedGenerator {
|
||||
fn generate_text(
|
||||
&self,
|
||||
req: GenerateRequest,
|
||||
) -> impl Future<Output = Result<GenerateResponse, String>> + Send {
|
||||
let i = {
|
||||
let mut calls = self.calls.lock().unwrap();
|
||||
let i = calls.len();
|
||||
calls.push(req.clone());
|
||||
i
|
||||
};
|
||||
let model = req.model.clone().unwrap_or_default();
|
||||
let entry = self.script.get(i)
|
||||
.cloned()
|
||||
.unwrap_or_else(|| self.script.last().cloned().unwrap_or(Ok(String::new())));
|
||||
async move {
|
||||
entry.map(|text| GenerateResponse {
|
||||
text,
|
||||
model,
|
||||
tokens_evaluated: None,
|
||||
tokens_generated: None,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn structural_complete_rejects_empty_and_text_mismatch() {
|
||||
assert!(!is_structurally_complete("", ResponseShape::Text));
|
||||
assert!(!is_structurally_complete(" ", ResponseShape::Text));
|
||||
assert!(is_structurally_complete("any content", ResponseShape::Text));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn structural_complete_handles_balanced_json() {
|
||||
assert!(is_structurally_complete(r#"{"a": 1}"#, ResponseShape::Json));
|
||||
assert!(is_structurally_complete(
|
||||
r#"```json
|
||||
{"a": 1, "b": [1, 2, 3]}
|
||||
```"#,
|
||||
ResponseShape::Json,
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn structural_complete_rejects_truncated_json() {
|
||||
assert!(!is_structurally_complete(r#"{"a": 1"#, ResponseShape::Json));
|
||||
assert!(!is_structurally_complete(r#"{"a": {"b": 1"#, ResponseShape::Json));
|
||||
// Trailing comma — balanced but unparseable
|
||||
assert!(!is_structurally_complete(r#"{"a": 1,}"#, ResponseShape::Json));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn structural_complete_ignores_braces_inside_strings() {
|
||||
assert!(is_structurally_complete(r#"{"s": "has } inside"}"#, ResponseShape::Json));
|
||||
assert!(is_structurally_complete(r#"{"s": "escaped \" quote"}"#, ResponseShape::Json));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn continuable_returns_first_response_when_complete() {
|
||||
let generator = ScriptedGenerator::new(vec![Ok(r#"{"ok": true}"#.to_string())]);
|
||||
let opts = ContinuableOpts::new("qwen3:latest");
|
||||
let out = generate_continuable(&generator, "test", &opts).await.unwrap();
|
||||
assert!(out.final_complete);
|
||||
assert_eq!(out.empty_retries, 0);
|
||||
assert_eq!(out.continuations, 0);
|
||||
assert_eq!(generator.call_count(), 1);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn continuable_retries_on_empty_with_doubled_budget() {
|
||||
// Two empties, then a good response. Third call should see 4×
|
||||
// the initial budget (2× twice).
|
||||
let generator = ScriptedGenerator::new(vec![
|
||||
Ok("".to_string()),
|
||||
Ok("".to_string()),
|
||||
Ok(r#"{"ok": true}"#.to_string()),
|
||||
]);
|
||||
let mut opts = ContinuableOpts::new("qwen3:latest");
|
||||
opts.max_tokens = Some(100);
|
||||
let out = generate_continuable(&generator, "test", &opts).await.unwrap();
|
||||
assert!(out.final_complete);
|
||||
assert_eq!(out.empty_retries, 2);
|
||||
|
||||
let calls = generator.calls();
|
||||
assert_eq!(calls.len(), 3);
|
||||
assert_eq!(calls[0].max_tokens, Some(100));
|
||||
assert_eq!(calls[1].max_tokens, Some(200));
|
||||
assert_eq!(calls[2].max_tokens, Some(400));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn continuable_caps_budget_at_budget_cap() {
|
||||
let generator = ScriptedGenerator::new(vec![
|
||||
Ok("".to_string()),
|
||||
Ok("".to_string()),
|
||||
Ok(r#"{"ok":1}"#.to_string()),
|
||||
]);
|
||||
let mut opts = ContinuableOpts::new("qwen3:latest");
|
||||
opts.max_tokens = Some(5_000);
|
||||
opts.budget_cap = 7_000;
|
||||
let out = generate_continuable(&generator, "test", &opts).await.unwrap();
|
||||
assert!(out.final_complete);
|
||||
let calls = generator.calls();
|
||||
// 5000 → doubled would be 10000; cap pulls it to 7000.
|
||||
assert_eq!(calls[1].max_tokens, Some(7_000));
|
||||
assert_eq!(calls[2].max_tokens, Some(7_000));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn continuable_glues_truncated_response() {
|
||||
// First call returns balanced-open `{...`; continuation closes
|
||||
// it with `...}`. Combined must parse.
|
||||
let generator = ScriptedGenerator::new(vec![
|
||||
Ok(r#"{"fills": [{"candidate_id": "C-001""#.to_string()),
|
||||
Ok(r#", "name": "Alice"}]}"#.to_string()),
|
||||
]);
|
||||
let opts = ContinuableOpts::new("qwen3:latest");
|
||||
let out = generate_continuable(&generator, "ORIGINAL", &opts).await.unwrap();
|
||||
assert!(out.final_complete, "combined must parse: {}", out.text);
|
||||
assert_eq!(out.continuations, 1);
|
||||
|
||||
let calls = generator.calls();
|
||||
assert_eq!(calls.len(), 2);
|
||||
// Continuation prompt must contain the partial — that's the
|
||||
// scratchpad primitive J called out.
|
||||
let cont_prompt = &calls[1].prompt;
|
||||
assert!(cont_prompt.contains("ORIGINAL"),
|
||||
"continuation must include original prompt");
|
||||
assert!(cont_prompt.contains(r#"{"fills": [{"candidate_id": "C-001""#),
|
||||
"continuation must include partial");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn continuable_does_not_loop_on_persistent_empty() {
|
||||
// All three retries return empty; we must NOT then enter the
|
||||
// continuation loop with an empty partial (would burn 3 more
|
||||
// calls continuing from "").
|
||||
let generator = ScriptedGenerator::new(vec![
|
||||
Ok("".to_string()),
|
||||
Ok("".to_string()),
|
||||
Ok("".to_string()),
|
||||
]);
|
||||
let opts = ContinuableOpts::new("qwen3:latest");
|
||||
let out = generate_continuable(&generator, "test", &opts).await.unwrap();
|
||||
assert!(!out.final_complete);
|
||||
assert_eq!(out.empty_retries, 3);
|
||||
assert_eq!(out.continuations, 0, "must not continue from empty");
|
||||
assert_eq!(generator.call_count(), 3);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn continuable_returns_raw_on_exhausted_continuations() {
|
||||
// Three continuations that never complete — caller's parser
|
||||
// will throw. We must return the combined text so forensics
|
||||
// has the raw content, not a lossy truncation.
|
||||
let generator = ScriptedGenerator::new(vec![
|
||||
Ok(r#"{"a": ["#.to_string()),
|
||||
Ok("1,".to_string()),
|
||||
Ok("2,".to_string()),
|
||||
Ok("3,".to_string()),
|
||||
]);
|
||||
let mut opts = ContinuableOpts::new("qwen3:latest");
|
||||
opts.max_continuations = 3;
|
||||
let out = generate_continuable(&generator, "test", &opts).await.unwrap();
|
||||
assert!(!out.final_complete);
|
||||
assert_eq!(out.continuations, 3);
|
||||
assert!(out.text.contains(r#"{"a": ["#));
|
||||
assert!(out.text.contains("3,"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn continuable_propagates_generator_errors() {
|
||||
let generator = ScriptedGenerator::new(vec![Err("sidecar 503".to_string())]);
|
||||
let opts = ContinuableOpts::new("qwen3:latest");
|
||||
let err = generate_continuable(&generator, "test", &opts).await.unwrap_err();
|
||||
assert!(err.contains("503"));
|
||||
}
|
||||
}
|
||||
@@ -1,2 +1,5 @@
|
||||
pub mod client;
|
||||
pub mod context;
|
||||
pub mod continuation;
|
||||
pub mod service;
|
||||
pub mod tree_split;
|
||||
|
||||
@@ -0,0 +1,326 @@
|
||||
//! Phase 21 — INPUT-overflow handler. Ports `generateTreeSplit` from
|
||||
//! `tests/multi-agent/agent.ts`.
|
||||
//!
|
||||
//! When the input corpus exceeds the model's window (200 playbooks
|
||||
//! pasted into a T4 strategic prompt, a long retrospective digest, a
|
||||
//! cross-corpus summarization), raising `max_tokens` doesn't help —
|
||||
//! the prompt itself is the problem. The answer is map-reduce:
|
||||
//!
|
||||
//! 1. Caller shards the input at semantic boundaries (records,
|
||||
//! paragraphs, playbook entries).
|
||||
//! 2. For each shard, build a map prompt that includes the running
|
||||
//! scratchpad and run it through `generate_continuable`.
|
||||
//! 3. Append the map output to the scratchpad (oldest-first
|
||||
//! truncation when it outgrows `scratchpad_budget`).
|
||||
//! 4. Build a reduce prompt from the final scratchpad and run it.
|
||||
//!
|
||||
//! Every shard prompt and the reduce prompt go through
|
||||
//! `assert_context_budget` first — if a single shard still overflows
|
||||
//! we bubble the error up rather than silently truncating. That's the
|
||||
//! whole point of Phase 21.
|
||||
|
||||
use crate::context::{assert_context_budget, BudgetOpts, estimate_tokens, overflow_message,
|
||||
DEFAULT_MAX_TOKENS, DEFAULT_SAFETY_MARGIN};
|
||||
use crate::continuation::{generate_continuable, ContinuableOpts, ResponseShape, TextGenerator};
|
||||
|
||||
/// Callback signatures — caller supplies closures that stitch the
|
||||
/// scratchpad into each shard's prompt and build the final reduce
|
||||
/// prompt. Kept as `Fn` (not `FnMut`) so the map loop can call them
|
||||
/// by reference.
|
||||
pub type MapPromptFn<'a> = dyn Fn(&str, &str) -> String + Send + Sync + 'a;
|
||||
pub type ReducePromptFn<'a> = dyn Fn(&str) -> String + Send + Sync + 'a;
|
||||
|
||||
/// Knobs for `generate_tree_split`.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct TreeSplitOpts {
|
||||
pub model: String,
|
||||
pub system: Option<String>,
|
||||
pub temperature: Option<f64>,
|
||||
/// max_tokens for map AND reduce (reduce defaults are usually
|
||||
/// higher; caller overrides for just reduce by calling through
|
||||
/// continuable directly if needed).
|
||||
pub max_tokens: Option<u32>,
|
||||
pub reduce_max_tokens: Option<u32>,
|
||||
pub think: Option<bool>,
|
||||
/// Soft ceiling on scratchpad size (estimated tokens). When it
|
||||
/// grows past this, the oldest shard digest gets dropped. Default
|
||||
/// 6000, matching the TS implementation.
|
||||
pub scratchpad_budget: usize,
|
||||
pub safety_margin: Option<usize>,
|
||||
}
|
||||
|
||||
impl TreeSplitOpts {
|
||||
pub fn new(model: impl Into<String>) -> Self {
|
||||
Self {
|
||||
model: model.into(),
|
||||
system: None,
|
||||
temperature: None,
|
||||
max_tokens: None,
|
||||
reduce_max_tokens: None,
|
||||
think: None,
|
||||
scratchpad_budget: 6_000,
|
||||
safety_margin: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Result — final reduce response plus the accumulated scratchpad so
|
||||
/// the caller can inspect what was kept vs truncated.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct TreeSplitResult {
|
||||
pub response: String,
|
||||
pub scratchpad: String,
|
||||
pub shards_processed: usize,
|
||||
pub scratchpad_truncations: usize,
|
||||
pub total_continuations: usize,
|
||||
}
|
||||
|
||||
/// Drop shard-digest blocks from the head of `scratchpad` until its
|
||||
/// estimated-token count fits the budget. Digest blocks are delimited
|
||||
/// by `\n— shard N/M digest —\n` so we can find the first one and
|
||||
/// chop everything before its successor.
|
||||
fn truncate_scratchpad(scratchpad: &mut String, budget_tokens: usize) -> bool {
|
||||
if estimate_tokens(scratchpad) <= budget_tokens { return false; }
|
||||
// Find the second delimiter — everything before it gets dropped.
|
||||
const DELIM_PREFIX: &str = "\n— shard ";
|
||||
let mut cursor = 0;
|
||||
let mut truncated = false;
|
||||
while estimate_tokens(&scratchpad[cursor..]) > budget_tokens {
|
||||
// Skip past a leading delimiter (if we're sitting on one from
|
||||
// a previous iteration), then find the next.
|
||||
let search_from = cursor + if scratchpad[cursor..].starts_with(DELIM_PREFIX) {
|
||||
DELIM_PREFIX.len()
|
||||
} else { 0 };
|
||||
let Some(rel_next) = scratchpad[search_from..].find(DELIM_PREFIX) else { break };
|
||||
cursor = search_from + rel_next;
|
||||
truncated = true;
|
||||
}
|
||||
if cursor > 0 {
|
||||
scratchpad.drain(..cursor);
|
||||
}
|
||||
truncated
|
||||
}
|
||||
|
||||
/// Phase 21 — map-reduce over shards with a running scratchpad. See
|
||||
/// module docs.
|
||||
pub async fn generate_tree_split<G: TextGenerator>(
|
||||
generator: &G,
|
||||
shards: &[String],
|
||||
map_prompt: &MapPromptFn<'_>,
|
||||
reduce_prompt: &ReducePromptFn<'_>,
|
||||
opts: &TreeSplitOpts,
|
||||
) -> Result<TreeSplitResult, String> {
|
||||
let mut scratchpad = String::new();
|
||||
let safety = opts.safety_margin.unwrap_or(DEFAULT_SAFETY_MARGIN);
|
||||
let map_max = opts.max_tokens.unwrap_or(DEFAULT_MAX_TOKENS as u32);
|
||||
let reduce_max = opts.reduce_max_tokens.unwrap_or(1_500);
|
||||
let mut truncations = 0usize;
|
||||
let mut total_continuations = 0usize;
|
||||
|
||||
for (i, shard) in shards.iter().enumerate() {
|
||||
let shard_prompt = map_prompt(shard, &scratchpad);
|
||||
// Loud-fail on per-shard overflow — caller sharded too
|
||||
// coarsely. Silent truncation is exactly the mode J rejected.
|
||||
let budget = BudgetOpts {
|
||||
system: opts.system.as_deref(),
|
||||
max_tokens: Some(map_max as usize),
|
||||
safety_margin: Some(safety),
|
||||
bypass: false,
|
||||
};
|
||||
let check = assert_context_budget(&opts.model, &shard_prompt, budget)
|
||||
.map_err(|(c, over)| overflow_message(&opts.model, &c, over, safety))?;
|
||||
let _ = check;
|
||||
|
||||
let mut cont_opts = ContinuableOpts::new(&opts.model);
|
||||
cont_opts.max_tokens = Some(map_max);
|
||||
cont_opts.temperature = opts.temperature;
|
||||
cont_opts.system = opts.system.clone();
|
||||
cont_opts.shape = ResponseShape::Text;
|
||||
cont_opts.think = opts.think;
|
||||
|
||||
let outcome = generate_continuable(generator, &shard_prompt, &cont_opts).await?;
|
||||
total_continuations += outcome.continuations;
|
||||
|
||||
// Append this shard's digest and, if needed, drop oldest.
|
||||
scratchpad.push_str(&format!(
|
||||
"\n— shard {}/{} digest —\n{}",
|
||||
i + 1, shards.len(), outcome.text.trim(),
|
||||
));
|
||||
if truncate_scratchpad(&mut scratchpad, opts.scratchpad_budget) {
|
||||
truncations += 1;
|
||||
}
|
||||
}
|
||||
|
||||
// Reduce pass. Budget check first — if the scratchpad is still too
|
||||
// big for the reduce prompt we fail loud with numbers.
|
||||
let reduce_p = reduce_prompt(&scratchpad);
|
||||
let budget = BudgetOpts {
|
||||
system: opts.system.as_deref(),
|
||||
max_tokens: Some(reduce_max as usize),
|
||||
safety_margin: Some(safety),
|
||||
bypass: false,
|
||||
};
|
||||
assert_context_budget(&opts.model, &reduce_p, budget)
|
||||
.map_err(|(c, over)| overflow_message(&opts.model, &c, over, safety))?;
|
||||
|
||||
let mut cont_opts = ContinuableOpts::new(&opts.model);
|
||||
cont_opts.max_tokens = Some(reduce_max);
|
||||
cont_opts.temperature = opts.temperature;
|
||||
cont_opts.system = opts.system.clone();
|
||||
cont_opts.shape = ResponseShape::Text;
|
||||
cont_opts.think = opts.think;
|
||||
|
||||
let outcome = generate_continuable(generator, &reduce_p, &cont_opts).await?;
|
||||
total_continuations += outcome.continuations;
|
||||
|
||||
Ok(TreeSplitResult {
|
||||
response: outcome.text,
|
||||
scratchpad,
|
||||
shards_processed: shards.len(),
|
||||
scratchpad_truncations: truncations,
|
||||
total_continuations,
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::continuation::ScriptedGenerator;
|
||||
|
||||
fn simple_map(shard: &str, scratchpad: &str) -> String {
|
||||
format!("SCRATCHPAD:\n{scratchpad}\n---\nSHARD:\n{shard}\n---\nDIGEST:")
|
||||
}
|
||||
|
||||
fn simple_reduce(scratchpad: &str) -> String {
|
||||
format!("SCRATCHPAD:\n{scratchpad}\n---\nFINAL:")
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn tree_split_runs_map_then_reduce() {
|
||||
// 3 shards → 3 map calls → 1 reduce call = 4 responses scripted.
|
||||
let generator = ScriptedGenerator::new(vec![
|
||||
Ok("digest-1".to_string()),
|
||||
Ok("digest-2".to_string()),
|
||||
Ok("digest-3".to_string()),
|
||||
Ok("FINAL ANSWER".to_string()),
|
||||
]);
|
||||
let shards: Vec<String> = ["a", "b", "c"].iter().map(|s| s.to_string()).collect();
|
||||
let opts = TreeSplitOpts::new("qwen3:latest");
|
||||
let map_fn: &MapPromptFn = &simple_map;
|
||||
let reduce_fn: &ReducePromptFn = &simple_reduce;
|
||||
let result = generate_tree_split(&generator, &shards, map_fn, reduce_fn, &opts)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(result.shards_processed, 3);
|
||||
assert_eq!(result.response, "FINAL ANSWER");
|
||||
assert_eq!(generator.call_count(), 4);
|
||||
// Scratchpad must carry all three digests in order.
|
||||
assert!(result.scratchpad.contains("digest-1"));
|
||||
assert!(result.scratchpad.contains("digest-2"));
|
||||
assert!(result.scratchpad.contains("digest-3"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn tree_split_reduce_prompt_sees_full_scratchpad() {
|
||||
let generator = ScriptedGenerator::new(vec![
|
||||
Ok("summary-A".to_string()),
|
||||
Ok("summary-B".to_string()),
|
||||
Ok("REDUCED".to_string()),
|
||||
]);
|
||||
let shards = vec!["input-one".to_string(), "input-two".to_string()];
|
||||
let opts = TreeSplitOpts::new("qwen3:latest");
|
||||
let _ = generate_tree_split(&generator, &shards, &simple_map, &simple_reduce, &opts)
|
||||
.await
|
||||
.unwrap();
|
||||
// Third call = reduce. Its prompt must include both digests.
|
||||
let calls = generator.calls();
|
||||
let reduce_prompt = &calls[2].prompt;
|
||||
assert!(reduce_prompt.contains("summary-A"),
|
||||
"reduce prompt must see first shard digest");
|
||||
assert!(reduce_prompt.contains("summary-B"),
|
||||
"reduce prompt must see second shard digest");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn tree_split_loud_fails_on_shard_overflow() {
|
||||
let generator = ScriptedGenerator::new(vec![Ok("digest".to_string())]);
|
||||
// One gigantic shard — well over qwen3's 40K window even as a
|
||||
// prompt. The budget check must reject before any generate call.
|
||||
let shards = vec!["x".repeat(200_000)];
|
||||
let opts = TreeSplitOpts::new("qwen3:latest");
|
||||
let err = generate_tree_split(&generator, &shards, &simple_map, &simple_reduce, &opts)
|
||||
.await
|
||||
.expect_err("shard-sized overflow must be rejected");
|
||||
assert!(err.contains("overflow"), "error should mention overflow: {err}");
|
||||
assert_eq!(generator.call_count(), 0, "generate must not be called on overflow");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn tree_split_truncates_scratchpad_when_over_budget() {
|
||||
// Tight budget so each shard trips truncation. qwen3's 40K
|
||||
// window is fine; the budget we care about is the scratchpad
|
||||
// cap, not the model window.
|
||||
let generator = ScriptedGenerator::new(vec![
|
||||
Ok("A".repeat(2_000)),
|
||||
Ok("B".repeat(2_000)),
|
||||
Ok("C".repeat(2_000)),
|
||||
Ok("D".repeat(2_000)),
|
||||
Ok("FINAL".to_string()),
|
||||
]);
|
||||
let shards: Vec<String> = (0..4).map(|i| format!("shard{i}")).collect();
|
||||
let mut opts = TreeSplitOpts::new("qwen3:latest");
|
||||
opts.scratchpad_budget = 1_000; // ~4000 chars — one digest barely fits
|
||||
let result = generate_tree_split(&generator, &shards, &simple_map, &simple_reduce, &opts)
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(result.scratchpad_truncations > 0,
|
||||
"tight budget must trigger truncation");
|
||||
// Scratchpad should still fit roughly within the budget
|
||||
// (post-truncation); the estimator uses chars/4 so the bound
|
||||
// is ~budget*4 chars. Give some slack for the delimiter.
|
||||
let scratchpad_tokens = estimate_tokens(&result.scratchpad);
|
||||
assert!(scratchpad_tokens <= opts.scratchpad_budget * 2,
|
||||
"scratchpad {} tokens vs budget {}", scratchpad_tokens, opts.scratchpad_budget);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn tree_split_reports_continuations_from_map_and_reduce() {
|
||||
// First shard: truncated-then-continued. Reduce: truncated-then-continued.
|
||||
// 1 shard: 2 map calls (initial + continuation), then 2 reduce calls.
|
||||
let generator = ScriptedGenerator::new(vec![
|
||||
Ok("partial".to_string()), // map shape=text, non-empty → complete on first pass
|
||||
Ok("reduce-out".to_string()),
|
||||
]);
|
||||
let shards = vec!["only".to_string()];
|
||||
let opts = TreeSplitOpts::new("qwen3:latest");
|
||||
let result = generate_tree_split(&generator, &shards, &simple_map, &simple_reduce, &opts)
|
||||
.await
|
||||
.unwrap();
|
||||
// Text shape treats non-empty as complete → 0 continuations.
|
||||
assert_eq!(result.total_continuations, 0);
|
||||
assert_eq!(result.shards_processed, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncate_scratchpad_noop_when_under_budget() {
|
||||
let mut s = "\n— shard 1/1 digest —\nshort".to_string();
|
||||
let truncated = truncate_scratchpad(&mut s, 1_000);
|
||||
assert!(!truncated);
|
||||
assert!(s.contains("short"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncate_scratchpad_drops_oldest_first() {
|
||||
let mut s = format!(
|
||||
"\n— shard 1/3 digest —\n{}\n— shard 2/3 digest —\n{}\n— shard 3/3 digest —\nshort",
|
||||
"x".repeat(4_000), // ~1000 tokens
|
||||
"y".repeat(4_000), // ~1000 tokens
|
||||
);
|
||||
let truncated = truncate_scratchpad(&mut s, 500); // ~2000 chars
|
||||
assert!(truncated);
|
||||
assert!(!s.contains(&"x".repeat(4_000)),
|
||||
"oldest digest should be dropped");
|
||||
assert!(s.contains("short"),
|
||||
"newest digest should survive");
|
||||
}
|
||||
}
|
||||
@@ -54,7 +54,16 @@ impl CatalogService for CatalogGrpc {
|
||||
let manifest = self.registry
|
||||
.register(req.name, SchemaFingerprint(req.schema_fingerprint), objects)
|
||||
.await
|
||||
.map_err(|e| Status::internal(e))?;
|
||||
.map_err(|e| {
|
||||
// Mirror the HTTP side's 409 mapping: schema drift is a
|
||||
// precondition failure, not a server error. Keeps gRPC
|
||||
// and HTTP callers seeing the same diagnostic signal.
|
||||
if e.contains("different schema") {
|
||||
Status::failed_precondition(e)
|
||||
} else {
|
||||
Status::internal(e)
|
||||
}
|
||||
})?;
|
||||
|
||||
Ok(Response::new(manifest_to_proto(&manifest)))
|
||||
}
|
||||
|
||||
@@ -28,6 +28,27 @@ pub struct MigrateBucketsReport {
|
||||
pub manifests_persisted: usize,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Default, serde::Serialize)]
|
||||
pub struct DedupeReport {
|
||||
pub groups: usize,
|
||||
pub removed: usize,
|
||||
pub kept: Vec<DedupeKept>,
|
||||
pub errors: Vec<DedupeError>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, serde::Serialize)]
|
||||
pub struct DedupeKept {
|
||||
pub name: String,
|
||||
pub kept_id: String,
|
||||
pub removed: usize,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, serde::Serialize)]
|
||||
pub struct DedupeError {
|
||||
pub manifest_id: String,
|
||||
pub error: String,
|
||||
}
|
||||
|
||||
/// Partial metadata update — only set fields are applied.
|
||||
#[derive(Debug, Clone, Default, serde::Deserialize)]
|
||||
pub struct MetadataUpdate {
|
||||
@@ -212,13 +233,65 @@ impl Registry {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Register a new dataset. Persists manifest to storage before updating memory.
|
||||
/// Register a dataset. Idempotent on `name`:
|
||||
/// - No existing dataset by this name → create new manifest.
|
||||
/// - Exists with same schema fingerprint → update objects + `updated_at`
|
||||
/// in place (re-ingest of identical-shape data).
|
||||
/// - Exists with different schema fingerprint → reject as `Err`
|
||||
/// (callers map to HTTP 409 / gRPC FAILED_PRECONDITION). Schema
|
||||
/// evolution is not yet exposed as an HTTP endpoint; callers must
|
||||
/// register under a new name or delete the existing manifest first.
|
||||
///
|
||||
/// Concurrency: the write lock is held across the storage write. That
|
||||
/// serializes concurrent registers but is correctness-mandatory —
|
||||
/// dropping the lock across the check→insert sequence creates a TOCTOU
|
||||
/// window where two callers with the same name both see "no existing"
|
||||
/// and both insert, reintroducing the duplicate-manifest bug this
|
||||
/// function exists to prevent. Register is a metadata-only hop on the
|
||||
/// ingest path, so the serialization cost is acceptable. Precedent:
|
||||
/// `update_metadata` also holds the write lock across its I/O.
|
||||
///
|
||||
/// Legacy state: if multiple manifests with the same name already exist
|
||||
/// (from before this was idempotent), the most recently updated one is
|
||||
/// treated as canonical. Run `dedupe_by_name` to clean up the rest.
|
||||
pub async fn register(
|
||||
&self,
|
||||
name: String,
|
||||
schema_fingerprint: SchemaFingerprint,
|
||||
objects: Vec<ObjectRef>,
|
||||
) -> Result<DatasetManifest, String> {
|
||||
let mut datasets = self.datasets.write().await;
|
||||
|
||||
let existing = datasets
|
||||
.values()
|
||||
.filter(|d| d.name == name)
|
||||
.max_by_key(|d| d.updated_at)
|
||||
.cloned();
|
||||
|
||||
if let Some(mut manifest) = existing {
|
||||
if manifest.schema_fingerprint != schema_fingerprint {
|
||||
return Err(format!(
|
||||
"dataset '{}' already exists with a different schema \
|
||||
(existing fingerprint: {}, new: {}). Schema-evolution \
|
||||
migration is not yet exposed as an HTTP endpoint; \
|
||||
register under a new name or delete the existing \
|
||||
manifest first.",
|
||||
name, manifest.schema_fingerprint.0, schema_fingerprint.0,
|
||||
));
|
||||
}
|
||||
manifest.objects = objects;
|
||||
manifest.updated_at = chrono::Utc::now();
|
||||
|
||||
let manifest_key = format!("{MANIFEST_PREFIX}/{}.json", manifest.id);
|
||||
let json = serde_json::to_vec_pretty(&manifest).map_err(|e| e.to_string())?;
|
||||
ops::put(&self.store, &manifest_key, json.into()).await?;
|
||||
|
||||
datasets.insert(manifest.id.clone(), manifest.clone());
|
||||
|
||||
tracing::info!("re-registered (idempotent): {} ({})", manifest.name, manifest.id);
|
||||
return Ok(manifest);
|
||||
}
|
||||
|
||||
let now = chrono::Utc::now();
|
||||
let manifest = DatasetManifest {
|
||||
id: DatasetId::new(),
|
||||
@@ -240,12 +313,10 @@ impl Registry {
|
||||
embedding_refresh_policy: None,
|
||||
};
|
||||
|
||||
// Write-ahead: persist before in-memory update
|
||||
let manifest_key = format!("{MANIFEST_PREFIX}/{}.json", manifest.id);
|
||||
let json = serde_json::to_vec_pretty(&manifest).map_err(|e| e.to_string())?;
|
||||
ops::put(&self.store, &manifest_key, json.into()).await?;
|
||||
|
||||
let mut datasets = self.datasets.write().await;
|
||||
datasets.insert(manifest.id.clone(), manifest.clone());
|
||||
|
||||
tracing::info!("registered dataset: {} ({})", manifest.name, manifest.id);
|
||||
@@ -380,6 +451,72 @@ impl Registry {
|
||||
Ok(manifest.clone())
|
||||
}
|
||||
|
||||
/// Collapse duplicate manifests that share a `name`. Winner per group:
|
||||
/// 1. Prefer a manifest with a non-null `row_count` (already resynced).
|
||||
/// 2. Break ties by newest `updated_at`.
|
||||
/// Losers are removed from the in-memory registry AND their manifest
|
||||
/// JSON is deleted from object storage. Datasets with a single manifest
|
||||
/// are untouched. Parquet data files are never touched — only catalog
|
||||
/// metadata. See `register` for the prevention side of this fix.
|
||||
///
|
||||
/// Operator-only. Do not run while ingest is active: a concurrent
|
||||
/// `register` between the snapshot and the delete sweep can create a
|
||||
/// new manifest under a name we just deduped, producing a transient
|
||||
/// count of 2 until the next dedupe run. Safe outside ingest windows.
|
||||
pub async fn dedupe_by_name(&self) -> DedupeReport {
|
||||
let groups: Vec<(String, Vec<DatasetManifest>)> = {
|
||||
let datasets = self.datasets.read().await;
|
||||
let mut by_name: HashMap<String, Vec<DatasetManifest>> = HashMap::new();
|
||||
for m in datasets.values() {
|
||||
by_name.entry(m.name.clone()).or_default().push(m.clone());
|
||||
}
|
||||
by_name.into_iter().filter(|(_, v)| v.len() > 1).collect()
|
||||
};
|
||||
|
||||
let mut report = DedupeReport::default();
|
||||
let mut to_delete: Vec<DatasetId> = Vec::new();
|
||||
|
||||
for (name, mut manifests) in groups {
|
||||
report.groups += 1;
|
||||
manifests.sort_by(|a, b| {
|
||||
b.row_count.is_some().cmp(&a.row_count.is_some())
|
||||
.then(b.updated_at.cmp(&a.updated_at))
|
||||
});
|
||||
let winner = &manifests[0];
|
||||
report.kept.push(DedupeKept {
|
||||
name: name.clone(),
|
||||
kept_id: winner.id.to_string(),
|
||||
removed: manifests.len() - 1,
|
||||
});
|
||||
for loser in &manifests[1..] {
|
||||
to_delete.push(loser.id.clone());
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
let mut datasets = self.datasets.write().await;
|
||||
for id in &to_delete {
|
||||
datasets.remove(id);
|
||||
}
|
||||
}
|
||||
for id in &to_delete {
|
||||
let key = format!("{MANIFEST_PREFIX}/{}.json", id);
|
||||
match ops::delete(&self.store, &key).await {
|
||||
Ok(_) => report.removed += 1,
|
||||
Err(e) => report.errors.push(DedupeError {
|
||||
manifest_id: id.to_string(),
|
||||
error: e,
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
tracing::info!(
|
||||
"dedupe: {} groups collapsed, {} manifests removed, {} errors",
|
||||
report.groups, report.removed, report.errors.len(),
|
||||
);
|
||||
report
|
||||
}
|
||||
|
||||
/// Resync every dataset that currently has a null row_count.
|
||||
/// Returns (successes, failures) where each entry is (name, detail).
|
||||
pub async fn resync_missing(&self) -> (Vec<(String, u64)>, Vec<(String, String)>) {
|
||||
@@ -549,6 +686,46 @@ impl Registry {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Remove a dataset from the catalog by name. Deletes the manifest
|
||||
/// from both the in-memory registry and object storage.
|
||||
///
|
||||
/// Scope is metadata-only: the underlying parquet files, vector
|
||||
/// indexes, tombstones, trial journals, and AiViews that reference
|
||||
/// this dataset are NOT touched. Caller is responsible for any
|
||||
/// cascade cleanup. This mirrors how a DROP TABLE in a typical
|
||||
/// warehouse separates "forget about this dataset" from "actually
|
||||
/// reclaim the bytes".
|
||||
///
|
||||
/// Returns Ok(()) when the dataset existed and was removed; an Err
|
||||
/// with "dataset not found" when no manifest by that name exists.
|
||||
/// Legacy state with duplicate names: all matching manifests are
|
||||
/// removed (effectively a post-hoc dedupe on that name).
|
||||
pub async fn delete_dataset(&self, name: &str) -> Result<usize, String> {
|
||||
let ids_to_remove: Vec<DatasetId> = {
|
||||
let datasets = self.datasets.read().await;
|
||||
datasets.values().filter(|d| d.name == name).map(|d| d.id.clone()).collect()
|
||||
};
|
||||
if ids_to_remove.is_empty() {
|
||||
return Err(format!("dataset not found: {name}"));
|
||||
}
|
||||
|
||||
for id in &ids_to_remove {
|
||||
let key = format!("{MANIFEST_PREFIX}/{}.json", id);
|
||||
if let Err(e) = ops::delete(&self.store, &key).await {
|
||||
// Storage delete failed — log and keep going. The in-memory
|
||||
// remove below is still correct; on restart, the missing
|
||||
// storage object just means nothing to rehydrate.
|
||||
tracing::warn!("delete_dataset '{name}': storage delete of {key} failed: {e}");
|
||||
}
|
||||
}
|
||||
let mut datasets = self.datasets.write().await;
|
||||
for id in &ids_to_remove {
|
||||
datasets.remove(id);
|
||||
}
|
||||
tracing::info!("deleted dataset '{}' ({} manifest(s))", name, ids_to_remove.len());
|
||||
Ok(ids_to_remove.len())
|
||||
}
|
||||
|
||||
/// List datasets whose `embedding_stale_since` is set — they need a refresh.
|
||||
pub async fn stale_datasets(&self) -> Vec<DatasetManifest> {
|
||||
let datasets = self.datasets.read().await;
|
||||
@@ -579,3 +756,275 @@ impl Registry {
|
||||
Ok(manifest.clone())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use object_store::memory::InMemory;
|
||||
|
||||
fn fixture() -> Registry {
|
||||
Registry::new(Arc::new(InMemory::new()))
|
||||
}
|
||||
|
||||
fn fp(s: &str) -> SchemaFingerprint {
|
||||
SchemaFingerprint(s.to_string())
|
||||
}
|
||||
|
||||
fn obj(key: &str) -> ObjectRef {
|
||||
ObjectRef {
|
||||
bucket: "primary".to_string(),
|
||||
key: key.to_string(),
|
||||
size_bytes: 100,
|
||||
created_at: chrono::Utc::now(),
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn fresh_register_creates_new_manifest() {
|
||||
let reg = fixture();
|
||||
let m = reg.register("a".into(), fp("f1"), vec![obj("a.parquet")]).await.unwrap();
|
||||
assert_eq!(m.name, "a");
|
||||
assert_eq!(m.schema_fingerprint, fp("f1"));
|
||||
assert_eq!(reg.list().await.len(), 1);
|
||||
assert_eq!(reg.get_by_name("a").await.unwrap().id, m.id);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn re_register_same_fingerprint_is_idempotent() {
|
||||
let reg = fixture();
|
||||
let first = reg.register("a".into(), fp("f1"), vec![obj("old.parquet")]).await.unwrap();
|
||||
let second = reg.register("a".into(), fp("f1"), vec![obj("new.parquet")]).await.unwrap();
|
||||
|
||||
assert_eq!(first.id, second.id, "same name+fp should reuse ID");
|
||||
assert_eq!(reg.list().await.len(), 1, "no duplicate manifest created");
|
||||
let fetched = reg.get_by_name("a").await.unwrap();
|
||||
assert_eq!(fetched.objects[0].key, "new.parquet", "objects replaced");
|
||||
assert!(fetched.updated_at >= first.updated_at, "updated_at bumped");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn re_register_different_fingerprint_rejects() {
|
||||
let reg = fixture();
|
||||
reg.register("a".into(), fp("f1"), vec![obj("a.parquet")]).await.unwrap();
|
||||
let err = reg.register("a".into(), fp("f2"), vec![obj("a.parquet")]).await.unwrap_err();
|
||||
|
||||
assert!(err.contains("different schema"), "error mentions schema drift: {err}");
|
||||
assert_eq!(reg.list().await.len(), 1, "failed register did not mutate state");
|
||||
assert_eq!(reg.get_by_name("a").await.unwrap().schema_fingerprint, fp("f1"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn delete_dataset_removes_manifest_and_in_memory_entry() {
|
||||
let reg = fixture();
|
||||
reg.register("to_delete".into(), fp("f1"), vec![obj("d.parquet")]).await.unwrap();
|
||||
reg.register("keepme".into(), fp("f2"), vec![obj("k.parquet")]).await.unwrap();
|
||||
assert_eq!(reg.list().await.len(), 2);
|
||||
|
||||
let removed = reg.delete_dataset("to_delete").await.unwrap();
|
||||
assert_eq!(removed, 1, "one manifest removed");
|
||||
assert!(reg.get_by_name("to_delete").await.is_none());
|
||||
assert!(reg.get_by_name("keepme").await.is_some(), "unrelated dataset untouched");
|
||||
assert_eq!(reg.list().await.len(), 1);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn delete_dataset_404s_on_unknown_name() {
|
||||
let reg = fixture();
|
||||
let err = reg.delete_dataset("nope").await.unwrap_err();
|
||||
assert!(err.starts_with("dataset not found"), "error wording: {err}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn delete_dataset_removes_all_dupes_with_matching_name() {
|
||||
// Legacy zombies: direct insert so we can seed duplicates,
|
||||
// mirroring pre-idempotent-register state. delete_dataset should
|
||||
// sweep every manifest sharing that name in a single call.
|
||||
let reg = fixture();
|
||||
let now = chrono::Utc::now();
|
||||
for _ in 0..3 {
|
||||
let m = DatasetManifest {
|
||||
id: DatasetId::new(),
|
||||
name: "zombie".into(),
|
||||
schema_fingerprint: fp("fp"),
|
||||
objects: vec![obj("z.parquet")],
|
||||
created_at: now, updated_at: now,
|
||||
row_count: None, description: String::new(), owner: String::new(),
|
||||
sensitivity: None, columns: vec![], lineage: None, freshness: None,
|
||||
tags: vec![], last_embedded_at: None, embedding_stale_since: None,
|
||||
embedding_refresh_policy: None,
|
||||
};
|
||||
reg.datasets.write().await.insert(m.id.clone(), m);
|
||||
}
|
||||
assert_eq!(reg.list().await.len(), 3);
|
||||
|
||||
let removed = reg.delete_dataset("zombie").await.unwrap();
|
||||
assert_eq!(removed, 3, "all three manifests removed in one sweep");
|
||||
assert_eq!(reg.list().await.len(), 0);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn dedupe_no_duplicates_is_noop() {
|
||||
let reg = fixture();
|
||||
reg.register("a".into(), fp("f1"), vec![obj("a.parquet")]).await.unwrap();
|
||||
reg.register("b".into(), fp("f2"), vec![obj("b.parquet")]).await.unwrap();
|
||||
|
||||
let report = reg.dedupe_by_name().await;
|
||||
assert_eq!(report.groups, 0);
|
||||
assert_eq!(report.removed, 0);
|
||||
assert_eq!(reg.list().await.len(), 2);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 8)]
|
||||
async fn concurrent_register_stress_many_workers_one_manifest() {
|
||||
// Stress variant: fire 32 concurrent registers for the same new
|
||||
// name across 8 worker threads. Single-manifest invariant must
|
||||
// hold regardless of how the scheduler interleaves them. This
|
||||
// catches race regressions the 2-call variant below might miss
|
||||
// under a forgiving scheduler.
|
||||
let reg = fixture();
|
||||
let mut handles = Vec::new();
|
||||
for i in 0..32 {
|
||||
let r = reg.clone();
|
||||
handles.push(tokio::spawn(async move {
|
||||
r.register(
|
||||
"stress".into(),
|
||||
fp("stress-fp"),
|
||||
vec![obj(&format!("{i}.parquet"))],
|
||||
).await
|
||||
}));
|
||||
}
|
||||
for h in handles {
|
||||
h.await.unwrap().expect("each register succeeds (idempotent)");
|
||||
}
|
||||
let all = reg.list().await;
|
||||
let stress_manifests: Vec<_> = all.iter().filter(|d| d.name == "stress").collect();
|
||||
assert_eq!(stress_manifests.len(), 1,
|
||||
"32 concurrent registers produced {} manifests for 'stress' — expected 1",
|
||||
stress_manifests.len());
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn concurrent_register_same_new_name_collapses_to_one_manifest() {
|
||||
// Pins the TOCTOU fix: without the write-lock held across the
|
||||
// check→insert sequence, two parallel registers of the same
|
||||
// previously-unknown name would both see "no existing" and both
|
||||
// insert. Multi-thread flavor is required to actually exercise
|
||||
// the race — under single-threaded tokio the scheduler doesn't
|
||||
// preempt between awaits on the same task's critical section.
|
||||
// Under the current impl, whichever acquires the write lock
|
||||
// first creates the manifest; the second observes it and takes
|
||||
// the idempotent-update branch.
|
||||
let reg = fixture();
|
||||
let a = reg.clone();
|
||||
let b = reg.clone();
|
||||
|
||||
let (ra, rb) = tokio::join!(
|
||||
async move {
|
||||
a.register("race".into(), fp("shared-fp"), vec![obj("a.parquet")]).await
|
||||
},
|
||||
async move {
|
||||
b.register("race".into(), fp("shared-fp"), vec![obj("b.parquet")]).await
|
||||
},
|
||||
);
|
||||
|
||||
let m_a = ra.expect("first register succeeds");
|
||||
let m_b = rb.expect("second register succeeds (same fp = idempotent)");
|
||||
assert_eq!(m_a.id, m_b.id, "both calls resolve to a single DatasetId");
|
||||
assert_eq!(reg.list().await.len(), 1, "no duplicate manifest created");
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn concurrent_register_same_name_different_fp_one_wins_one_rejects() {
|
||||
// Reverse case: concurrent registers with conflicting fingerprints.
|
||||
// Deterministic outcome: one call's fingerprint gets persisted as
|
||||
// canonical, the other is rejected on seeing the now-established
|
||||
// fingerprint. Which one wins is scheduler-dependent — we only
|
||||
// assert exactly one success and exactly one rejection.
|
||||
let reg = fixture();
|
||||
let a = reg.clone();
|
||||
let b = reg.clone();
|
||||
|
||||
let (ra, rb) = tokio::join!(
|
||||
async move { a.register("race".into(), fp("fp1"), vec![obj("a.parquet")]).await },
|
||||
async move { b.register("race".into(), fp("fp2"), vec![obj("b.parquet")]).await },
|
||||
);
|
||||
|
||||
let successes = [&ra, &rb].iter().filter(|r| r.is_ok()).count();
|
||||
let rejections = [&ra, &rb].iter().filter(|r| r.is_err()).count();
|
||||
assert_eq!(successes, 1, "exactly one register wins");
|
||||
assert_eq!(rejections, 1, "the other is rejected on fingerprint");
|
||||
let rejection_msg = [&ra, &rb].iter()
|
||||
.find_map(|r| r.as_ref().err())
|
||||
.unwrap();
|
||||
assert!(rejection_msg.contains("different schema"));
|
||||
assert_eq!(reg.list().await.len(), 1);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn dedupe_collapses_dupes_preferring_non_null_row_count() {
|
||||
let reg = fixture();
|
||||
|
||||
// Simulate legacy pre-idempotent-register state by inserting
|
||||
// manifests with the same `name` directly. Mirrors what happened
|
||||
// with the 308× successful_playbooks duplication.
|
||||
let now = chrono::Utc::now();
|
||||
let old_with_rows = DatasetManifest {
|
||||
id: DatasetId::new(),
|
||||
name: "dupes".into(),
|
||||
schema_fingerprint: fp("f1"),
|
||||
objects: vec![obj("dupes.parquet")],
|
||||
created_at: now - chrono::Duration::hours(2),
|
||||
updated_at: now - chrono::Duration::hours(2),
|
||||
row_count: Some(42),
|
||||
description: String::new(),
|
||||
owner: String::new(),
|
||||
sensitivity: None,
|
||||
columns: vec![],
|
||||
lineage: None,
|
||||
freshness: None,
|
||||
tags: vec![],
|
||||
last_embedded_at: None,
|
||||
embedding_stale_since: None,
|
||||
embedding_refresh_policy: None,
|
||||
};
|
||||
let newer_no_rows = DatasetManifest {
|
||||
id: DatasetId::new(),
|
||||
updated_at: now, // newer, but null row_count
|
||||
row_count: None,
|
||||
..old_with_rows.clone()
|
||||
};
|
||||
let oldest = DatasetManifest {
|
||||
id: DatasetId::new(),
|
||||
updated_at: now - chrono::Duration::hours(5),
|
||||
row_count: None,
|
||||
..old_with_rows.clone()
|
||||
};
|
||||
|
||||
let kept_id = old_with_rows.id.clone();
|
||||
let loser_a = newer_no_rows.id.clone();
|
||||
let loser_b = oldest.id.clone();
|
||||
|
||||
{
|
||||
let mut d = reg.datasets.write().await;
|
||||
d.insert(old_with_rows.id.clone(), old_with_rows);
|
||||
d.insert(newer_no_rows.id.clone(), newer_no_rows);
|
||||
d.insert(oldest.id.clone(), oldest);
|
||||
}
|
||||
assert_eq!(reg.list().await.len(), 3);
|
||||
|
||||
let report = reg.dedupe_by_name().await;
|
||||
assert_eq!(report.groups, 1);
|
||||
assert_eq!(report.removed, 2);
|
||||
assert_eq!(report.errors.len(), 0);
|
||||
assert_eq!(report.kept.len(), 1);
|
||||
assert_eq!(report.kept[0].name, "dupes");
|
||||
assert_eq!(report.kept[0].kept_id, kept_id.to_string(),
|
||||
"should keep the manifest with non-null row_count");
|
||||
|
||||
let remaining = reg.list().await;
|
||||
assert_eq!(remaining.len(), 1);
|
||||
assert_eq!(remaining[0].id, kept_id);
|
||||
assert!(reg.get(&loser_a).await.is_none());
|
||||
assert!(reg.get(&loser_b).await.is_none());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -17,11 +17,12 @@ pub fn router(registry: Registry) -> Router {
|
||||
.route("/datasets", post(create_dataset))
|
||||
.route("/datasets", get(list_datasets))
|
||||
.route("/datasets/{id}", get(get_dataset))
|
||||
.route("/datasets/by-name/{name}", get(get_dataset_by_name))
|
||||
.route("/datasets/by-name/{name}", get(get_dataset_by_name).delete(delete_dataset_by_name))
|
||||
.route("/datasets/by-name/{name}/metadata", post(update_metadata))
|
||||
.route("/datasets/by-name/{name}/resync", post(resync_dataset))
|
||||
.route("/resync-missing", post(resync_all_missing))
|
||||
.route("/migrate-buckets", post(migrate_buckets))
|
||||
.route("/dedupe", post(dedupe_by_name))
|
||||
// Phase D: AI-safe views
|
||||
.route("/views", post(create_view).get(list_views))
|
||||
.route("/views/{name}", get(get_view).delete(delete_view))
|
||||
@@ -121,7 +122,14 @@ async fn create_dataset(
|
||||
let resp = DatasetResponse::from(&manifest);
|
||||
Ok((StatusCode::CREATED, Json(resp)))
|
||||
}
|
||||
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
|
||||
Err(e) => {
|
||||
let status = if e.contains("different schema") {
|
||||
StatusCode::CONFLICT
|
||||
} else {
|
||||
StatusCode::INTERNAL_SERVER_ERROR
|
||||
};
|
||||
Err((status, e))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -153,6 +161,25 @@ async fn get_dataset_by_name(
|
||||
}
|
||||
}
|
||||
|
||||
/// Remove a dataset manifest by name. Metadata only — parquet files,
|
||||
/// vector indexes, and tombstones are NOT cascade-deleted. See
|
||||
/// `Registry::delete_dataset` for the full scope.
|
||||
async fn delete_dataset_by_name(
|
||||
State(registry): State<Registry>,
|
||||
Path(name): Path<String>,
|
||||
) -> impl IntoResponse {
|
||||
match registry.delete_dataset(&name).await {
|
||||
Ok(removed) => {
|
||||
let body = serde_json::json!({ "name": name, "manifests_removed": removed });
|
||||
Ok(Json(body))
|
||||
}
|
||||
Err(e) if e.starts_with("dataset not found") => {
|
||||
Err((StatusCode::NOT_FOUND, e))
|
||||
}
|
||||
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
|
||||
}
|
||||
}
|
||||
|
||||
async fn update_metadata(
|
||||
State(registry): State<Registry>,
|
||||
Path(name): Path<String>,
|
||||
@@ -214,6 +241,12 @@ async fn migrate_buckets(State(registry): State<Registry>) -> impl IntoResponse
|
||||
}
|
||||
}
|
||||
|
||||
/// Collapse duplicate manifests by `name` — see `Registry::dedupe_by_name`.
|
||||
/// Safe to run repeatedly; single-manifest datasets are untouched.
|
||||
async fn dedupe_by_name(State(registry): State<Registry>) -> impl IntoResponse {
|
||||
Json(registry.dedupe_by_name().await)
|
||||
}
|
||||
|
||||
// --- Phase D: AI-safe views ---
|
||||
|
||||
#[derive(Deserialize)]
|
||||
|
||||
@@ -28,3 +28,4 @@ opentelemetry-stdout = { workspace = true }
|
||||
tracing-opentelemetry = { workspace = true }
|
||||
arrow = { workspace = true }
|
||||
chrono = { workspace = true }
|
||||
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
|
||||
|
||||
@@ -3,6 +3,7 @@ mod access_service;
|
||||
mod auth;
|
||||
mod observability;
|
||||
mod tools;
|
||||
mod v1;
|
||||
|
||||
use axum::{Router, extract::DefaultBodyLimit, routing::get};
|
||||
use proto::lakehouse::catalog_service_server::CatalogServiceServer;
|
||||
@@ -86,6 +87,11 @@ async fn main() {
|
||||
// primary. Journal / registry resolve per-call via the bucket registry.
|
||||
let tj = vectord::trial::TrialJournal::new(bucket_registry.clone(), index_reg.clone());
|
||||
let pr = vectord::promotion::PromotionRegistry::new(bucket_registry.clone(), index_reg.clone());
|
||||
let hs = vectord::harness::HarnessStore::new(bucket_registry.clone(), index_reg.clone());
|
||||
// Phase 19: playbook memory. Load cached state; empty-on-miss is fine,
|
||||
// operators call POST /vectors/playbook_memory/rebuild to populate.
|
||||
let pbm = vectord::playbook_memory::PlaybookMemory::new(store.clone());
|
||||
let _ = pbm.load_from_storage().await;
|
||||
|
||||
// Phase 16.2: spawn the autotune agent. When config.agent.enabled=false
|
||||
// this returns a handle that drops triggers silently — no surprise load.
|
||||
@@ -107,6 +113,7 @@ async fn main() {
|
||||
embedding_cache: emb_cache.clone(),
|
||||
trial_journal: tj.clone(),
|
||||
promotion_registry: pr.clone(),
|
||||
harness_store: hs.clone(),
|
||||
},
|
||||
);
|
||||
|
||||
@@ -155,6 +162,7 @@ async fn main() {
|
||||
hnsw_store: hnsw,
|
||||
embedding_cache: emb_cache,
|
||||
trial_journal: tj,
|
||||
harness_store: hs,
|
||||
catalog: registry.clone(),
|
||||
promotion_registry: pr,
|
||||
agent_handle,
|
||||
@@ -163,6 +171,8 @@ async fn main() {
|
||||
lance: vectord::lance_backend::LanceRegistry::new(
|
||||
bucket_registry.clone(), index_reg.clone(),
|
||||
),
|
||||
playbook_memory: pbm,
|
||||
embed_semaphore: std::sync::Arc::new(tokio::sync::Semaphore::new(1)),
|
||||
}))
|
||||
.nest("/workspaces", queryd::workspace_service::router(workspace_mgr))
|
||||
.nest("/journal", journald::service::router(journal))
|
||||
@@ -174,6 +184,38 @@ async fn main() {
|
||||
registry: tool_reg,
|
||||
query_fn: tools::QueryExecutor::new(engine.clone()),
|
||||
}
|
||||
}))
|
||||
// Phase 38 — Universal API skeleton. Thin OpenAI-compatible
|
||||
// surface over the existing aibridge → Ollama path. Future
|
||||
// phases add provider adapters (39), routing engine (40),
|
||||
// session state (41), etc. All without changing this mount.
|
||||
.nest("/v1", v1::router(v1::V1State {
|
||||
ai_client: ai_client.clone(),
|
||||
usage: std::sync::Arc::new(tokio::sync::RwLock::new(v1::Usage::default())),
|
||||
// Phase 39 first slice — Ollama Cloud adapter. Key resolved
|
||||
// from OLLAMA_CLOUD_KEY env, then /root/llm_team_config.json,
|
||||
// then OLLAMA_CLOUD_API_KEY env. None = cloud routes 503.
|
||||
ollama_cloud_key: {
|
||||
let k = v1::ollama_cloud::resolve_cloud_key();
|
||||
if k.is_some() {
|
||||
tracing::info!("v1: Ollama Cloud key loaded — /v1/chat provider=ollama_cloud enabled");
|
||||
} else {
|
||||
tracing::warn!("v1: no Ollama Cloud key in env or /root/llm_team_config.json — cloud routes will 503");
|
||||
}
|
||||
k
|
||||
},
|
||||
// Phase 40 early deliverable — Langfuse trace emitter.
|
||||
// Defaults match mcp-server/tracing.ts conventions so
|
||||
// gateway traces land in the same staffing project.
|
||||
langfuse: {
|
||||
let c = v1::langfuse_trace::LangfuseClient::from_env_or_defaults();
|
||||
if c.is_some() {
|
||||
tracing::info!("v1: Langfuse tracing enabled — /v1/chat calls will appear at localhost:3001");
|
||||
} else {
|
||||
tracing::warn!("v1: Langfuse keys missing — /v1/chat calls will not be traced");
|
||||
}
|
||||
c
|
||||
},
|
||||
}));
|
||||
|
||||
// Auth middleware (if enabled)
|
||||
@@ -190,7 +232,7 @@ async fn main() {
|
||||
}
|
||||
|
||||
app = app
|
||||
.layer(DefaultBodyLimit::max(256 * 1024 * 1024)) // 256MB
|
||||
.layer(DefaultBodyLimit::max(512 * 1024 * 1024)) // 512MB — supports 500K worker CSV
|
||||
.layer(CorsLayer::new()
|
||||
.allow_origin(Any)
|
||||
.allow_methods(Any)
|
||||
|
||||
@@ -103,7 +103,7 @@ impl ToolRegistry {
|
||||
ParamDef { name: "limit".into(), param_type: "integer".into(), required: false, description: "Max results".into(), default: Some(serde_json::json!(20)) },
|
||||
],
|
||||
returns: "List of candidates with id, name, phone, email, skills, experience".into(),
|
||||
sql_template: "SELECT candidate_id, first_name, last_name, phone, email, city, state, zip, vertical, skills, years_experience FROM candidates WHERE 1=1 {skills_filter} {city_filter} {state_filter} {years_filter} {status_filter} ORDER BY years_experience DESC LIMIT {limit}".into(),
|
||||
sql_template: "SELECT candidate_id, first_name, last_name, phone, email, city, state, skills, years_experience, status FROM candidates WHERE 1=1 {skills_filter} {city_filter} {state_filter} {years_filter} {status_filter} ORDER BY years_experience DESC LIMIT {limit}".into(),
|
||||
category: "candidates".into(),
|
||||
},
|
||||
ToolDef {
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
//! Phase 40 (early deliverable) — Langfuse tracing from the Rust gateway.
|
||||
//!
|
||||
//! Langfuse is already running at `http://localhost:3001` with the
|
||||
//! staffing project — `mcp-server/tracing.ts` has been emitting traces
|
||||
//! for months. This file bridges the gap: every `/v1/chat` call now
|
||||
//! appears in the same Langfuse project, alongside traces from the
|
||||
//! Bun side.
|
||||
//!
|
||||
//! Design:
|
||||
//! - Fire-and-forget — emit spawns a tokio task. Response to client
|
||||
//! never waits on Langfuse, never fails because of it. If Langfuse
|
||||
//! is down or slow, we silently drop traces (and `tracing::warn!`).
|
||||
//! - Two events per call — `trace-create` + `generation-create`. The
|
||||
//! generation nests under the trace so Langfuse UI shows the whole
|
||||
//! thing as one clickable row.
|
||||
//! - No SDK — plain HTTP POST to `/api/public/ingestion` with Basic
|
||||
//! auth. Keeps the Rust dependency graph small.
|
||||
|
||||
use serde::Serialize;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use super::Message;
|
||||
|
||||
const INGESTION_PATH: &str = "/api/public/ingestion";
|
||||
const EMIT_TIMEOUT_SECS: u64 = 5;
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct LangfuseClient {
|
||||
inner: Arc<Inner>,
|
||||
}
|
||||
|
||||
struct Inner {
|
||||
base_url: String,
|
||||
public_key: String,
|
||||
secret_key: String,
|
||||
http: reqwest::Client,
|
||||
}
|
||||
|
||||
impl LangfuseClient {
|
||||
pub fn new(base_url: String, public_key: String, secret_key: String) -> Self {
|
||||
let http = reqwest::Client::builder()
|
||||
.timeout(Duration::from_secs(EMIT_TIMEOUT_SECS))
|
||||
.build()
|
||||
.expect("langfuse client init");
|
||||
Self { inner: Arc::new(Inner { base_url, public_key, secret_key, http }) }
|
||||
}
|
||||
|
||||
/// Resolve a LangfuseClient from the same conventions the TS side
|
||||
/// already uses. Returns None if keys are missing — caller logs a
|
||||
/// warning and omits tracing (non-fatal).
|
||||
///
|
||||
/// Sources checked in order:
|
||||
/// 1. LANGFUSE_URL + LANGFUSE_PUBLIC_KEY + LANGFUSE_SECRET_KEY env
|
||||
/// 2. /root/llm_team_config.json (if it carries langfuse keys)
|
||||
/// 3. Hardcoded staffing-project defaults matching mcp-server
|
||||
/// (pk-lf-staffing / sk-lf-staffing-secret / localhost:3001)
|
||||
pub fn from_env_or_defaults() -> Option<Self> {
|
||||
let base = std::env::var("LANGFUSE_URL")
|
||||
.unwrap_or_else(|_| "http://localhost:3001".to_string());
|
||||
let pk = std::env::var("LANGFUSE_PUBLIC_KEY")
|
||||
.unwrap_or_else(|_| "pk-lf-staffing".to_string());
|
||||
let sk = std::env::var("LANGFUSE_SECRET_KEY")
|
||||
.unwrap_or_else(|_| "sk-lf-staffing-secret".to_string());
|
||||
if pk.trim().is_empty() || sk.trim().is_empty() { return None; }
|
||||
Some(Self::new(base, pk, sk))
|
||||
}
|
||||
|
||||
/// Fire-and-forget emit. Never blocks, never errors upward.
|
||||
pub fn emit_chat(&self, ev: ChatTrace) {
|
||||
let this = self.clone();
|
||||
tokio::spawn(async move {
|
||||
if let Err(e) = this.emit_chat_inner(ev).await {
|
||||
tracing::warn!(target: "v1.langfuse", "trace drop: {e}");
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
async fn emit_chat_inner(&self, ev: ChatTrace) -> Result<(), String> {
|
||||
let trace_id = uuid_v7_like();
|
||||
let gen_id = uuid_v7_like();
|
||||
let trace_ts = ev.start_time.clone();
|
||||
|
||||
let batch = IngestionBatch {
|
||||
batch: vec![
|
||||
IngestionEvent {
|
||||
id: uuid_v7_like(),
|
||||
timestamp: trace_ts.clone(),
|
||||
kind: "trace-create",
|
||||
body: serde_json::json!({
|
||||
"id": trace_id,
|
||||
"name": format!("v1.chat:{}", ev.provider),
|
||||
"input": serde_json::json!({
|
||||
"model": ev.model,
|
||||
"messages": ev.input,
|
||||
}),
|
||||
"metadata": serde_json::json!({
|
||||
"provider": ev.provider,
|
||||
"think": ev.think,
|
||||
}),
|
||||
}),
|
||||
},
|
||||
IngestionEvent {
|
||||
id: uuid_v7_like(),
|
||||
timestamp: ev.end_time.clone(),
|
||||
kind: "generation-create",
|
||||
body: serde_json::json!({
|
||||
"id": gen_id,
|
||||
"traceId": trace_id,
|
||||
"name": "chat",
|
||||
"model": ev.model,
|
||||
"modelParameters": serde_json::json!({
|
||||
"temperature": ev.temperature,
|
||||
"max_tokens": ev.max_tokens,
|
||||
"think": ev.think,
|
||||
}),
|
||||
"input": ev.input,
|
||||
"output": ev.output,
|
||||
"usage": serde_json::json!({
|
||||
"input": ev.prompt_tokens,
|
||||
"output": ev.completion_tokens,
|
||||
"total": ev.prompt_tokens + ev.completion_tokens,
|
||||
"unit": "TOKENS",
|
||||
}),
|
||||
"startTime": ev.start_time,
|
||||
"endTime": ev.end_time,
|
||||
"metadata": serde_json::json!({
|
||||
"provider": ev.provider,
|
||||
"latency_ms": ev.latency_ms,
|
||||
}),
|
||||
}),
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
let url = format!("{}{}", self.inner.base_url.trim_end_matches('/'), INGESTION_PATH);
|
||||
let resp = self.inner.http
|
||||
.post(url)
|
||||
.basic_auth(&self.inner.public_key, Some(&self.inner.secret_key))
|
||||
.json(&batch)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| format!("POST failed: {e}"))?;
|
||||
if !resp.status().is_success() {
|
||||
return Err(format!("{}: {}", resp.status(), resp.text().await.unwrap_or_default()));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// Everything the v1.chat handler collects for one completed call.
|
||||
pub struct ChatTrace {
|
||||
pub provider: String,
|
||||
pub model: String,
|
||||
pub input: Vec<Message>,
|
||||
pub output: String,
|
||||
pub prompt_tokens: u32,
|
||||
pub completion_tokens: u32,
|
||||
pub temperature: Option<f64>,
|
||||
pub max_tokens: Option<u32>,
|
||||
pub think: Option<bool>,
|
||||
pub start_time: String,
|
||||
pub end_time: String,
|
||||
pub latency_ms: u64,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct IngestionBatch {
|
||||
batch: Vec<IngestionEvent>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct IngestionEvent {
|
||||
id: String,
|
||||
timestamp: String,
|
||||
#[serde(rename = "type")]
|
||||
kind: &'static str,
|
||||
body: serde_json::Value,
|
||||
}
|
||||
|
||||
/// UUIDv7-ish identifier — not strictly UUIDv7 but time-ordered and
|
||||
/// unique enough for event deduplication. Avoids a uuid crate dep.
|
||||
fn uuid_v7_like() -> String {
|
||||
let ts = chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0);
|
||||
let rand = std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.map(|d| d.subsec_nanos())
|
||||
.unwrap_or(0);
|
||||
format!("{:016x}-{:08x}", ts, rand)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn from_env_or_defaults_returns_some_when_defaults_present() {
|
||||
// Defaults are hardcoded non-empty — should always produce Some.
|
||||
let c = LangfuseClient::from_env_or_defaults();
|
||||
assert!(c.is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn uuid_v7_like_produces_distinct_ids() {
|
||||
let a = uuid_v7_like();
|
||||
// Different nanosecond timestamps — but they might collide on
|
||||
// back-to-back calls. Loop a bit to ensure uniqueness surface.
|
||||
let b = uuid_v7_like();
|
||||
let c = uuid_v7_like();
|
||||
assert!(a != b || b != c, "three consecutive ids should have at least one distinct pair");
|
||||
assert!(a.contains('-'));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ingestion_batch_serializes_with_rename() {
|
||||
let ev = IngestionEvent {
|
||||
id: "e1".into(),
|
||||
timestamp: "2026-04-22T00:00:00Z".into(),
|
||||
kind: "trace-create",
|
||||
body: serde_json::json!({"name": "t"}),
|
||||
};
|
||||
let json = serde_json::to_string(&IngestionBatch { batch: vec![ev] }).unwrap();
|
||||
assert!(json.contains("\"type\":\"trace-create\""));
|
||||
assert!(json.contains("\"batch\":["));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,256 @@
|
||||
//! Phase 38 — Universal API skeleton (`/v1/*`).
|
||||
//!
|
||||
//! OpenAI-compatible shape on top of the existing aibridge → Ollama
|
||||
//! path. This is the thin slice: single provider, stateless, no
|
||||
//! streaming. Phase 39 replaces the direct Ollama call with a
|
||||
//! `ProviderAdapter` trait dispatch; Phase 40 adds routing + fallback.
|
||||
//!
|
||||
//! The shape matches OpenAI's `/v1/chat/completions` closely enough
|
||||
//! that clients using openai-compatible SDKs can point at us with the
|
||||
//! URL swap alone. We keep the endpoint path `/v1/chat` (not
|
||||
//! `/v1/chat/completions`) because our PRD declares the terser form;
|
||||
//! adding the alias is one line in Phase 39 when it matters.
|
||||
|
||||
pub mod ollama;
|
||||
pub mod ollama_cloud;
|
||||
pub mod langfuse_trace;
|
||||
|
||||
use axum::{
|
||||
Router,
|
||||
extract::State,
|
||||
http::StatusCode,
|
||||
response::IntoResponse,
|
||||
routing::{get, post},
|
||||
Json,
|
||||
};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::sync::Arc;
|
||||
use tokio::sync::RwLock;
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct V1State {
|
||||
pub ai_client: aibridge::client::AiClient,
|
||||
pub usage: Arc<RwLock<Usage>>,
|
||||
/// Ollama Cloud bearer token. Loaded at startup via
|
||||
/// `ollama_cloud::resolve_cloud_key()`. None = cloud routes 503.
|
||||
pub ollama_cloud_key: Option<String>,
|
||||
/// Phase 40 early deliverable — Langfuse client. None = tracing
|
||||
/// disabled (keys missing or container unreachable). Traces are
|
||||
/// fire-and-forget: never block the response path.
|
||||
pub langfuse: Option<langfuse_trace::LangfuseClient>,
|
||||
}
|
||||
|
||||
#[derive(Default, Clone, Serialize)]
|
||||
pub struct Usage {
|
||||
pub requests: u64,
|
||||
pub prompt_tokens: u64,
|
||||
pub completion_tokens: u64,
|
||||
pub total_tokens: u64,
|
||||
}
|
||||
|
||||
pub fn router(state: V1State) -> Router {
|
||||
Router::new()
|
||||
.route("/chat", post(chat))
|
||||
.route("/usage", get(usage))
|
||||
.route("/sessions", get(sessions))
|
||||
.with_state(state)
|
||||
}
|
||||
|
||||
// -- Shared types (OpenAI-compatible) --
|
||||
|
||||
#[derive(Serialize, Deserialize, Clone, Debug)]
|
||||
pub struct Message {
|
||||
pub role: String,
|
||||
pub content: String,
|
||||
}
|
||||
|
||||
#[derive(Deserialize, Debug)]
|
||||
pub struct ChatRequest {
|
||||
pub model: String,
|
||||
pub messages: Vec<Message>,
|
||||
#[serde(default)]
|
||||
pub temperature: Option<f64>,
|
||||
#[serde(default)]
|
||||
pub max_tokens: Option<u32>,
|
||||
/// Accepted for shape-compat but ignored in the thin slice —
|
||||
/// Phase 38 returns non-streaming even when the client asked for it.
|
||||
/// Phase 39+ wires real streaming.
|
||||
#[serde(default)]
|
||||
pub stream: Option<bool>,
|
||||
/// Non-OpenAI extension. Passes through to the provider's thinking
|
||||
/// toggle. Default: **false** — hot-path discipline for thinking
|
||||
/// models (qwen3.5, qwen3, gpt-oss) that otherwise burn the token
|
||||
/// budget on hidden reasoning before visible output starts,
|
||||
/// producing empty responses. Set true explicitly when calling an
|
||||
/// overseer / reasoning-heavy path.
|
||||
#[serde(default)]
|
||||
pub think: Option<bool>,
|
||||
/// Non-OpenAI extension. Selects the provider adapter. Accepted:
|
||||
/// - None / "ollama" / "local" → local Ollama via sidecar (default)
|
||||
/// - "ollama_cloud" / "cloud" → Ollama Cloud direct HTTPS
|
||||
/// Phase 40 adds a routing engine that picks this automatically
|
||||
/// from the model name; Phase 38/39 requires explicit selection.
|
||||
#[serde(default)]
|
||||
pub provider: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct ChatResponse {
|
||||
pub id: String,
|
||||
pub object: &'static str,
|
||||
pub created: i64,
|
||||
pub model: String,
|
||||
pub choices: Vec<Choice>,
|
||||
pub usage: UsageBlock,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct Choice {
|
||||
pub index: u32,
|
||||
pub message: Message,
|
||||
pub finish_reason: String,
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize, Clone)]
|
||||
pub struct UsageBlock {
|
||||
pub prompt_tokens: u32,
|
||||
pub completion_tokens: u32,
|
||||
pub total_tokens: u32,
|
||||
}
|
||||
|
||||
// -- Handlers --
|
||||
|
||||
async fn chat(
|
||||
State(state): State<V1State>,
|
||||
Json(req): Json<ChatRequest>,
|
||||
) -> Result<Json<ChatResponse>, (StatusCode, String)> {
|
||||
if req.messages.is_empty() {
|
||||
return Err((StatusCode::BAD_REQUEST, "messages must be non-empty".into()));
|
||||
}
|
||||
if req.stream.unwrap_or(false) {
|
||||
tracing::warn!("/v1/chat: stream=true requested but Phase 38 returns non-streaming");
|
||||
}
|
||||
|
||||
let provider = req.provider.as_deref().unwrap_or("ollama").to_ascii_lowercase();
|
||||
let start_time = chrono::Utc::now();
|
||||
let start_instant = std::time::Instant::now();
|
||||
|
||||
let resp = match provider.as_str() {
|
||||
"ollama" | "local" | "" => ollama::chat(&state.ai_client, &req)
|
||||
.await
|
||||
.map_err(|e| (StatusCode::BAD_GATEWAY, format!("ollama local: {e}")))?,
|
||||
"ollama_cloud" | "cloud" => {
|
||||
let key = state.ollama_cloud_key.as_deref().ok_or((
|
||||
StatusCode::SERVICE_UNAVAILABLE,
|
||||
"OLLAMA_CLOUD_KEY not configured — set env or populate /root/llm_team_config.json".to_string(),
|
||||
))?;
|
||||
ollama_cloud::chat(key, &req)
|
||||
.await
|
||||
.map_err(|e| (StatusCode::BAD_GATEWAY, format!("ollama cloud: {e}")))?
|
||||
}
|
||||
other => {
|
||||
return Err((
|
||||
StatusCode::BAD_REQUEST,
|
||||
format!("unknown provider '{other}' — supported: ollama, ollama_cloud"),
|
||||
));
|
||||
}
|
||||
};
|
||||
|
||||
let end_time = chrono::Utc::now();
|
||||
let latency_ms = start_instant.elapsed().as_millis() as u64;
|
||||
|
||||
// Phase 40 — emit Langfuse trace. Fire-and-forget: the clone is
|
||||
// cheap (Arc inside), the tokio::spawn never blocks us, a dead
|
||||
// Langfuse just logs a warn. Client-visible response latency is
|
||||
// untouched.
|
||||
if let Some(lf) = &state.langfuse {
|
||||
let output = resp.choices.first()
|
||||
.map(|c| c.message.content.clone())
|
||||
.unwrap_or_default();
|
||||
lf.emit_chat(langfuse_trace::ChatTrace {
|
||||
provider: provider.clone(),
|
||||
model: resp.model.clone(),
|
||||
input: req.messages.clone(),
|
||||
output,
|
||||
prompt_tokens: resp.usage.prompt_tokens,
|
||||
completion_tokens: resp.usage.completion_tokens,
|
||||
temperature: req.temperature,
|
||||
max_tokens: req.max_tokens,
|
||||
think: req.think,
|
||||
start_time: start_time.to_rfc3339(),
|
||||
end_time: end_time.to_rfc3339(),
|
||||
latency_ms,
|
||||
});
|
||||
}
|
||||
|
||||
{
|
||||
let mut u = state.usage.write().await;
|
||||
u.requests += 1;
|
||||
u.prompt_tokens += resp.usage.prompt_tokens as u64;
|
||||
u.completion_tokens += resp.usage.completion_tokens as u64;
|
||||
u.total_tokens += resp.usage.total_tokens as u64;
|
||||
}
|
||||
|
||||
Ok(Json(resp))
|
||||
}
|
||||
|
||||
async fn usage(State(state): State<V1State>) -> impl IntoResponse {
|
||||
let snapshot = state.usage.read().await.clone();
|
||||
Json(snapshot)
|
||||
}
|
||||
|
||||
// Phase 38 is stateless — no session persistence yet. Return an empty
|
||||
// list in OpenAI-ish shape so clients that probe this endpoint don't
|
||||
// 404. Real session state lands in Phase 41 with the profile-system
|
||||
// expansion.
|
||||
async fn sessions() -> impl IntoResponse {
|
||||
Json(serde_json::json!({
|
||||
"data": [],
|
||||
"object": "list",
|
||||
"note": "Phase 38: stateless. Session state lands in Phase 41.",
|
||||
}))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn chat_request_parses_openai_shape() {
|
||||
let raw = r#"{
|
||||
"model": "qwen3.5:latest",
|
||||
"messages": [
|
||||
{"role": "system", "content": "You are helpful."},
|
||||
{"role": "user", "content": "Hi"}
|
||||
],
|
||||
"temperature": 0.2,
|
||||
"max_tokens": 100
|
||||
}"#;
|
||||
let r: ChatRequest = serde_json::from_str(raw).unwrap();
|
||||
assert_eq!(r.model, "qwen3.5:latest");
|
||||
assert_eq!(r.messages.len(), 2);
|
||||
assert_eq!(r.messages[0].role, "system");
|
||||
assert_eq!(r.messages[1].content, "Hi");
|
||||
assert_eq!(r.temperature, Some(0.2));
|
||||
assert_eq!(r.max_tokens, Some(100));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn chat_request_accepts_minimal() {
|
||||
let raw = r#"{
|
||||
"model": "any",
|
||||
"messages": [{"role": "user", "content": "hi"}]
|
||||
}"#;
|
||||
let r: ChatRequest = serde_json::from_str(raw).unwrap();
|
||||
assert_eq!(r.temperature, None);
|
||||
assert_eq!(r.max_tokens, None);
|
||||
assert_eq!(r.stream, None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn usage_counter_default_is_zero() {
|
||||
let u = Usage::default();
|
||||
assert_eq!(u.requests, 0);
|
||||
assert_eq!(u.total_tokens, 0);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
//! Phase 38 — Ollama shape adapter.
|
||||
//!
|
||||
//! Translates `/v1/chat` (OpenAI-compatible) requests into the
|
||||
//! existing aibridge `GenerateRequest` shape, and the `GenerateResponse`
|
||||
//! back into an OpenAI-compatible `ChatResponse`. This is a bridge,
|
||||
//! not a new client — aibridge + the Python sidecar stay as-is.
|
||||
//!
|
||||
//! Phase 39 replaces this direct call with a `ProviderAdapter` trait
|
||||
//! dispatch so the same `/v1/chat` handler routes to any provider.
|
||||
|
||||
use aibridge::client::{AiClient, GenerateRequest};
|
||||
use super::{ChatRequest, ChatResponse, Choice, Message, UsageBlock};
|
||||
|
||||
pub async fn chat(client: &AiClient, req: &ChatRequest) -> Result<ChatResponse, String> {
|
||||
let (system, prompt) = flatten_messages(&req.messages);
|
||||
|
||||
let gen_req = GenerateRequest {
|
||||
prompt,
|
||||
model: Some(req.model.clone()),
|
||||
system: if system.is_empty() { None } else { Some(system) },
|
||||
temperature: req.temperature,
|
||||
max_tokens: req.max_tokens,
|
||||
// Phase 38 default: think=false. Hot-path discipline for
|
||||
// thinking models (qwen3.5, qwen3, gpt-oss) which otherwise
|
||||
// burn the max_tokens budget on hidden reasoning before any
|
||||
// visible output, producing empty responses. Callers that
|
||||
// actually want reasoning (overseers, T3+ tiers) opt in via
|
||||
// the `think: true` extension field. Phase 40 routing engine
|
||||
// flips this per task class.
|
||||
think: Some(req.think.unwrap_or(false)),
|
||||
};
|
||||
|
||||
let t0 = std::time::Instant::now();
|
||||
let resp = client.generate(gen_req).await.map_err(|e| e.to_string())?;
|
||||
let latency_ms = t0.elapsed().as_millis();
|
||||
|
||||
// Prefer sidecar-reported token counts when present. Fall back to
|
||||
// chars/4 estimate (biased safe ~15%, matches Phase 21 convention
|
||||
// in crates/aibridge/src/context.rs::estimate_tokens).
|
||||
let prompt_tokens = resp.tokens_evaluated
|
||||
.map(|n| n as u32)
|
||||
.unwrap_or_else(|| estimate_prompt_tokens(&req.messages));
|
||||
let completion_tokens = resp.tokens_generated
|
||||
.map(|n| n as u32)
|
||||
.unwrap_or_else(|| ((resp.text.chars().count() + 3) / 4) as u32);
|
||||
|
||||
tracing::info!(
|
||||
target: "v1.chat",
|
||||
model = %req.model,
|
||||
prompt_tokens,
|
||||
completion_tokens,
|
||||
latency_ms = latency_ms as u64,
|
||||
"ollama chat completed",
|
||||
);
|
||||
|
||||
Ok(ChatResponse {
|
||||
id: format!("chatcmpl-{}", chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0)),
|
||||
object: "chat.completion",
|
||||
created: chrono::Utc::now().timestamp(),
|
||||
model: resp.model,
|
||||
choices: vec![Choice {
|
||||
index: 0,
|
||||
message: Message {
|
||||
role: "assistant".into(),
|
||||
content: resp.text,
|
||||
},
|
||||
finish_reason: "stop".into(),
|
||||
}],
|
||||
usage: UsageBlock {
|
||||
prompt_tokens,
|
||||
completion_tokens,
|
||||
total_tokens: prompt_tokens + completion_tokens,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
/// Public re-export of the flattener so sibling adapters (Ollama Cloud,
|
||||
/// future OpenRouter) can reuse the same shape collapse without
|
||||
/// duplicating the logic. Keeps `(system, prompt)` format consistent
|
||||
/// across providers.
|
||||
pub fn flatten_messages_public(messages: &[Message]) -> (String, String) {
|
||||
flatten_messages(messages)
|
||||
}
|
||||
|
||||
/// Collapse a message array into (system, prompt). Multiple system
|
||||
/// messages concatenate with a newline — matches OpenAI's documented
|
||||
/// behavior. Non-system messages become role-labeled turns.
|
||||
fn flatten_messages(messages: &[Message]) -> (String, String) {
|
||||
let mut system = String::new();
|
||||
let mut prompt = String::new();
|
||||
for m in messages {
|
||||
if m.role == "system" {
|
||||
if !system.is_empty() { system.push('\n'); }
|
||||
system.push_str(&m.content);
|
||||
} else {
|
||||
prompt.push_str(&m.role);
|
||||
prompt.push_str(": ");
|
||||
prompt.push_str(&m.content);
|
||||
prompt.push_str("\n\n");
|
||||
}
|
||||
}
|
||||
prompt.push_str("assistant:");
|
||||
(system, prompt)
|
||||
}
|
||||
|
||||
fn estimate_prompt_tokens(messages: &[Message]) -> u32 {
|
||||
let chars: usize = messages.iter().map(|m| m.content.chars().count()).sum();
|
||||
((chars + 3) / 4) as u32
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn flatten_separates_system_from_turns() {
|
||||
let msgs = vec![
|
||||
Message { role: "system".into(), content: "Rules here.".into() },
|
||||
Message { role: "user".into(), content: "Q1".into() },
|
||||
Message { role: "assistant".into(), content: "A1".into() },
|
||||
Message { role: "user".into(), content: "Q2".into() },
|
||||
];
|
||||
let (system, prompt) = flatten_messages(&msgs);
|
||||
assert_eq!(system, "Rules here.");
|
||||
assert!(prompt.contains("user: Q1"));
|
||||
assert!(prompt.contains("assistant: A1"));
|
||||
assert!(prompt.contains("user: Q2"));
|
||||
assert!(prompt.trim_end().ends_with("assistant:"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flatten_concatenates_multiple_system_messages() {
|
||||
let msgs = vec![
|
||||
Message { role: "system".into(), content: "First.".into() },
|
||||
Message { role: "system".into(), content: "Second.".into() },
|
||||
Message { role: "user".into(), content: "Hi".into() },
|
||||
];
|
||||
let (system, _) = flatten_messages(&msgs);
|
||||
assert_eq!(system, "First.\nSecond.");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flatten_with_no_system_returns_empty_system() {
|
||||
let msgs = vec![Message { role: "user".into(), content: "hi".into() }];
|
||||
let (system, prompt) = flatten_messages(&msgs);
|
||||
assert!(system.is_empty());
|
||||
assert!(prompt.contains("user: hi"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn estimate_tokens_chars_div_4_ceiling() {
|
||||
let msgs = vec![Message { role: "user".into(), content: "abcdefgh".into() }];
|
||||
// 8 chars / 4 = 2, with ceiling → 2
|
||||
assert_eq!(estimate_prompt_tokens(&msgs), 2);
|
||||
|
||||
let msgs2 = vec![Message { role: "user".into(), content: "abcdefghi".into() }];
|
||||
// 9 chars → (9+3)/4 = 3 (ceiling)
|
||||
assert_eq!(estimate_prompt_tokens(&msgs2), 3);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,189 @@
|
||||
//! Phase 39 (first slice) — Ollama Cloud adapter.
|
||||
//!
|
||||
//! Direct HTTPS call to `https://ollama.com/api/generate` with Bearer
|
||||
//! auth. Mirrors the `tests/multi-agent/agent.ts::generateCloud` pattern
|
||||
//! exactly (same endpoint, same body shape, same Bearer header), so
|
||||
//! cloud calls from Rust behave identically to the TS hot path.
|
||||
//!
|
||||
//! Key sourcing priority:
|
||||
//! 1. Env var `OLLAMA_CLOUD_KEY` (matches agent.ts convention)
|
||||
//! 2. `/root/llm_team_config.json` → providers.ollama_cloud.api_key
|
||||
//! 3. Env var `OLLAMA_CLOUD_API_KEY` (LLM Team UI convention)
|
||||
//!
|
||||
//! First hit wins. Key is loaded once at gateway startup by
|
||||
//! `resolve_cloud_key()` and stored on `V1State.ollama_cloud_key`.
|
||||
|
||||
use std::time::Duration;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use super::{ChatRequest, ChatResponse, Choice, Message, UsageBlock};
|
||||
|
||||
const CLOUD_BASE_URL: &str = "https://ollama.com";
|
||||
const CLOUD_TIMEOUT_SECS: u64 = 180;
|
||||
|
||||
/// Read the Ollama Cloud key from the three sanctioned sources. Returns
|
||||
/// None if none is set — callers must 503 rather than attempt a call.
|
||||
pub fn resolve_cloud_key() -> Option<String> {
|
||||
if let Ok(k) = std::env::var("OLLAMA_CLOUD_KEY") {
|
||||
if !k.trim().is_empty() { return Some(k.trim().to_string()); }
|
||||
}
|
||||
if let Ok(raw) = std::fs::read_to_string("/root/llm_team_config.json") {
|
||||
if let Ok(v) = serde_json::from_str::<serde_json::Value>(&raw) {
|
||||
if let Some(k) = v.pointer("/providers/ollama_cloud/api_key").and_then(|x| x.as_str()) {
|
||||
if !k.trim().is_empty() { return Some(k.trim().to_string()); }
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Ok(k) = std::env::var("OLLAMA_CLOUD_API_KEY") {
|
||||
if !k.trim().is_empty() { return Some(k.trim().to_string()); }
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
pub async fn chat(
|
||||
key: &str,
|
||||
req: &ChatRequest,
|
||||
) -> Result<ChatResponse, String> {
|
||||
let (system, prompt) = super::ollama::flatten_messages_public(&req.messages);
|
||||
|
||||
let body = CloudGenerateBody {
|
||||
model: req.model.clone(),
|
||||
prompt,
|
||||
system: if system.is_empty() { None } else { Some(system) },
|
||||
stream: false,
|
||||
think: Some(req.think.unwrap_or(false)),
|
||||
options: CloudOptions {
|
||||
// Thinking cloud models need headroom — floor 400 to give
|
||||
// qwen3.5:397b / gpt-oss:120b reasoning room. Matches
|
||||
// agent.ts `Math.max(opts.max_tokens ?? 800, 400)` policy.
|
||||
num_predict: std::cmp::max(req.max_tokens.unwrap_or(800), 400),
|
||||
temperature: req.temperature.unwrap_or(0.3),
|
||||
},
|
||||
};
|
||||
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(Duration::from_secs(CLOUD_TIMEOUT_SECS))
|
||||
.build()
|
||||
.map_err(|e| format!("build client: {e}"))?;
|
||||
|
||||
let t0 = std::time::Instant::now();
|
||||
let resp = client
|
||||
.post(format!("{}/api/generate", CLOUD_BASE_URL))
|
||||
.bearer_auth(key)
|
||||
.json(&body)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| format!("ollama.com unreachable: {e}"))?;
|
||||
|
||||
let status = resp.status();
|
||||
if !status.is_success() {
|
||||
let body = resp.text().await.unwrap_or_else(|_| "?".into());
|
||||
return Err(format!("ollama.com {}: {}", status, body));
|
||||
}
|
||||
|
||||
let parsed: CloudGenerateResponse = resp.json().await
|
||||
.map_err(|e| format!("invalid cloud response: {e}"))?;
|
||||
|
||||
let latency_ms = t0.elapsed().as_millis();
|
||||
let text = parsed.response.unwrap_or_default();
|
||||
|
||||
let prompt_tokens = parsed.prompt_eval_count.unwrap_or_else(|| {
|
||||
let chars: usize = req.messages.iter().map(|m| m.content.chars().count()).sum();
|
||||
((chars + 3) / 4) as u32
|
||||
});
|
||||
let completion_tokens = parsed.eval_count.unwrap_or_else(|| {
|
||||
((text.chars().count() + 3) / 4) as u32
|
||||
});
|
||||
|
||||
tracing::info!(
|
||||
target: "v1.chat",
|
||||
provider = "ollama_cloud",
|
||||
model = %req.model,
|
||||
prompt_tokens,
|
||||
completion_tokens,
|
||||
latency_ms = latency_ms as u64,
|
||||
"cloud chat completed",
|
||||
);
|
||||
|
||||
Ok(ChatResponse {
|
||||
id: format!("chatcmpl-{}", chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0)),
|
||||
object: "chat.completion",
|
||||
created: chrono::Utc::now().timestamp(),
|
||||
model: parsed.model.unwrap_or_else(|| req.model.clone()),
|
||||
choices: vec![Choice {
|
||||
index: 0,
|
||||
message: Message { role: "assistant".into(), content: text },
|
||||
finish_reason: "stop".into(),
|
||||
}],
|
||||
usage: UsageBlock {
|
||||
prompt_tokens,
|
||||
completion_tokens,
|
||||
total_tokens: prompt_tokens + completion_tokens,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
// -- Ollama /api/generate wire shapes --
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct CloudGenerateBody {
|
||||
model: String,
|
||||
prompt: String,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
system: Option<String>,
|
||||
stream: bool,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
think: Option<bool>,
|
||||
options: CloudOptions,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct CloudOptions {
|
||||
num_predict: u32,
|
||||
temperature: f64,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct CloudGenerateResponse {
|
||||
#[serde(default)]
|
||||
response: Option<String>,
|
||||
#[serde(default)]
|
||||
model: Option<String>,
|
||||
#[serde(default)]
|
||||
prompt_eval_count: Option<u32>,
|
||||
#[serde(default)]
|
||||
eval_count: Option<u32>,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn resolve_cloud_key_returns_none_when_no_sources_set() {
|
||||
// Only check shape — we can't reliably unset env vars in a test
|
||||
// that runs alongside others, and the file path is on disk.
|
||||
// If all three sources are empty the function returns None; if
|
||||
// any are set we expect Some. This just smoke-tests the call
|
||||
// doesn't panic.
|
||||
let _ = resolve_cloud_key();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cloud_body_serializes_compact() {
|
||||
let body = CloudGenerateBody {
|
||||
model: "gpt-oss:120b".into(),
|
||||
prompt: "user: hi\n\nassistant:".into(),
|
||||
system: Some("Be terse.".into()),
|
||||
stream: false,
|
||||
think: Some(false),
|
||||
options: CloudOptions { num_predict: 400, temperature: 0.3 },
|
||||
};
|
||||
let json = serde_json::to_string(&body).unwrap();
|
||||
assert!(json.contains("\"model\":\"gpt-oss:120b\""));
|
||||
assert!(json.contains("\"stream\":false"));
|
||||
assert!(json.contains("\"num_predict\":400"));
|
||||
assert!(json.contains("\"think\":false"));
|
||||
assert!(json.contains("\"system\":\"Be terse.\""));
|
||||
}
|
||||
}
|
||||
@@ -24,3 +24,4 @@ object_store = { workspace = true }
|
||||
tokio-postgres = { workspace = true }
|
||||
mysql_async = { workspace = true }
|
||||
uuid = { workspace = true }
|
||||
croner = "3"
|
||||
|
||||
+112
-13
@@ -17,8 +17,8 @@
|
||||
//! shouldn't pile up runs). Different schedules run concurrently.
|
||||
//!
|
||||
//! What's deliberately not in scope here:
|
||||
//! - Cron expressions (the trigger enum has the variant but parsing is
|
||||
//! stubbed). Intervals cover 90% of operational scheduling; cron is
|
||||
//! - Sub-minute cron precision. 5-field Unix cron is supported; seconds
|
||||
//! are pinned to 0. Intervals cover sub-minute cases; cron is
|
||||
//! easy to bolt on later.
|
||||
//! - Backoff / retry policies. A failed run records the failure and
|
||||
//! schedules `next_run_at` as if it succeeded — no exponential backoff.
|
||||
@@ -85,8 +85,12 @@ impl ScheduleKind {
|
||||
pub enum ScheduleTrigger {
|
||||
/// Run every N seconds. Time anchor = compute_next_run_at decision.
|
||||
Interval { secs: u64 },
|
||||
/// Cron expression — parsing not implemented yet. Defining the
|
||||
/// variant now so the JSON shape is forward-compatible.
|
||||
/// 5-field Unix cron expression: "min hour dom month dow".
|
||||
/// Seconds are implicitly 0 (minute-granularity scheduling).
|
||||
/// Examples: "15 14 * * *" = 14:15 UTC daily, "0 */6 * * *" = every
|
||||
/// 6 hours on the hour, "0 9 * * 1-5" = 09:00 UTC weekdays.
|
||||
/// Timezone: UTC always — the scheduler's clock is UTC and we don't
|
||||
/// carry a per-schedule timezone field.
|
||||
Cron { expr: String },
|
||||
}
|
||||
|
||||
@@ -162,22 +166,55 @@ impl ScheduleDef {
|
||||
// Parse the expression, find the next match after `now`. Most
|
||||
// expressive but pulls in a cron parser dep.
|
||||
//
|
||||
// Cron is currently stubbed — falling back to "now + 1 hour" so a
|
||||
// mistakenly-configured Cron schedule doesn't melt the GPU.
|
||||
pub fn compute_next_run_at(
|
||||
trigger: &ScheduleTrigger,
|
||||
completed_at: DateTime<Utc>,
|
||||
_previous_next: DateTime<Utc>,
|
||||
) -> DateTime<Utc> {
|
||||
// TODO(J): pick a strategy. Starter = since-last-completion.
|
||||
match trigger {
|
||||
ScheduleTrigger::Interval { secs } => {
|
||||
completed_at + Duration::seconds(*secs as i64)
|
||||
}
|
||||
ScheduleTrigger::Cron { expr: _ } => {
|
||||
// Cron parsing not implemented — fall back to a safe 1h
|
||||
// window so a bad config can't fire-loop the system.
|
||||
completed_at + Duration::hours(1)
|
||||
ScheduleTrigger::Cron { expr } => {
|
||||
next_cron_fire(expr, completed_at)
|
||||
// Creation-time validation (see `validate_trigger`) rejects
|
||||
// unparseable expressions, so this fallback is only reached
|
||||
// if the schedule JSON was hand-edited on disk. One-hour
|
||||
// window keeps a bad expr from fire-looping the system.
|
||||
.unwrap_or(completed_at + Duration::hours(1))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Parse a Vixie/POSIX cron expression and return the next fire time
|
||||
/// strictly after `after`. Accepts 5-field (`min hour dom month dow`) and
|
||||
/// 6-field (with leading `sec`) forms natively — croner handles both.
|
||||
/// Day-of-week follows Unix convention: 0=Sun, 1=Mon, …, 6=Sat (7=Sun as
|
||||
/// synonym). Returns None if the expression fails to parse or produces no
|
||||
/// upcoming match (which can happen for year-bounded patterns in the past).
|
||||
fn next_cron_fire(expr: &str, after: DateTime<Utc>) -> Option<DateTime<Utc>> {
|
||||
use std::str::FromStr;
|
||||
let cron = croner::Cron::from_str(expr).ok()?;
|
||||
cron.find_next_occurrence(&after, false).ok()
|
||||
}
|
||||
|
||||
/// Reject a ScheduleTrigger whose content can't be interpreted — used by
|
||||
/// the HTTP handlers to fail fast at create/patch time rather than falling
|
||||
/// back silently at fire time. Interval triggers get a sanity gate
|
||||
/// (no zero secs); cron triggers get a full parse through croner.
|
||||
pub fn validate_trigger(trigger: &ScheduleTrigger) -> Result<(), String> {
|
||||
match trigger {
|
||||
ScheduleTrigger::Interval { secs } => {
|
||||
if *secs == 0 {
|
||||
return Err("interval secs must be > 0".into());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
ScheduleTrigger::Cron { expr } => {
|
||||
use std::str::FromStr;
|
||||
croner::Cron::from_str(expr)
|
||||
.map_err(|e| format!("invalid cron expression '{expr}': {e}"))?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -551,6 +588,7 @@ fn redact_dsn(dsn: &str) -> String {
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use chrono::Timelike;
|
||||
|
||||
fn mk(secs: u64) -> ScheduleDef {
|
||||
ScheduleDef {
|
||||
@@ -600,12 +638,73 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cron_falls_back_to_one_hour() {
|
||||
fn cron_every_minute_fires_within_sixty_seconds() {
|
||||
let trig = ScheduleTrigger::Cron { expr: "* * * * *".into() };
|
||||
let now = Utc::now();
|
||||
let next = compute_next_run_at(&trig, now, now);
|
||||
let delta = next - now;
|
||||
assert_eq!(delta.num_seconds(), 3600);
|
||||
assert!(delta.num_seconds() > 0 && delta.num_seconds() <= 60,
|
||||
"expected next fire within 60s, got {}s", delta.num_seconds());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cron_daily_at_1415_utc_is_within_24h() {
|
||||
// 14:15 UTC daily — whether we're before or after 14:15 today, the
|
||||
// next fire is at most 24h out.
|
||||
let trig = ScheduleTrigger::Cron { expr: "15 14 * * *".into() };
|
||||
let now: DateTime<Utc> = "2026-04-20T10:00:00Z".parse().unwrap();
|
||||
let next = compute_next_run_at(&trig, now, now);
|
||||
assert!(next > now);
|
||||
assert!((next - now).num_hours() <= 24);
|
||||
// And the minute-of-hour is 15.
|
||||
assert_eq!(next.minute(), 15);
|
||||
assert_eq!(next.hour(), 14);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cron_weekday_skips_weekend() {
|
||||
// 09:00 UTC weekdays only. On a Saturday, next fire is Monday.
|
||||
let trig = ScheduleTrigger::Cron { expr: "0 9 * * 1-5".into() };
|
||||
// 2026-04-18 is a Saturday.
|
||||
let sat: DateTime<Utc> = "2026-04-18T10:00:00Z".parse().unwrap();
|
||||
let next = compute_next_run_at(&trig, sat, sat);
|
||||
// Monday 2026-04-20 at 09:00 UTC.
|
||||
assert_eq!(next, "2026-04-20T09:00:00Z".parse::<DateTime<Utc>>().unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cron_six_field_seconds_granularity() {
|
||||
// 6-field (seconds included): fire every minute at 30s past the minute.
|
||||
let trig = ScheduleTrigger::Cron { expr: "30 * * * * *".into() };
|
||||
let now = Utc::now();
|
||||
let next = compute_next_run_at(&trig, now, now);
|
||||
assert_eq!(next.second(), 30);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn validate_rejects_bad_cron() {
|
||||
let bad = ScheduleTrigger::Cron { expr: "not a cron".into() };
|
||||
let err = validate_trigger(&bad).unwrap_err();
|
||||
assert!(err.contains("cron"), "error should mention cron, got: {err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn validate_rejects_wrong_field_count() {
|
||||
// 4 fields — neither 5 nor 6.
|
||||
let bad = ScheduleTrigger::Cron { expr: "* * * *".into() };
|
||||
assert!(validate_trigger(&bad).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn validate_rejects_zero_interval() {
|
||||
let bad = ScheduleTrigger::Interval { secs: 0 };
|
||||
assert!(validate_trigger(&bad).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn validate_accepts_good_cron() {
|
||||
let good = ScheduleTrigger::Cron { expr: "0 */6 * * *".into() };
|
||||
assert!(validate_trigger(&good).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -510,6 +510,9 @@ async fn create_schedule(
|
||||
if state.schedules.get(&req.id).await.is_some() {
|
||||
return Err((StatusCode::CONFLICT, format!("schedule '{}' already exists", req.id)));
|
||||
}
|
||||
if let Err(e) = schedule::validate_trigger(&req.trigger) {
|
||||
return Err((StatusCode::BAD_REQUEST, e));
|
||||
}
|
||||
let now = chrono::Utc::now();
|
||||
let def = schedule::ScheduleDef {
|
||||
id: req.id,
|
||||
@@ -564,6 +567,11 @@ async fn patch_schedule(
|
||||
let Some(mut def) = state.schedules.get(&id).await else {
|
||||
return Err((StatusCode::NOT_FOUND, format!("schedule '{id}' not found")));
|
||||
};
|
||||
if let Some(t) = &req.trigger {
|
||||
if let Err(e) = schedule::validate_trigger(t) {
|
||||
return Err((StatusCode::BAD_REQUEST, e));
|
||||
}
|
||||
}
|
||||
if let Some(e) = req.enabled { def.enabled = e; }
|
||||
if let Some(t) = req.trigger { def.trigger = t; }
|
||||
match state.schedules.put(def).await {
|
||||
|
||||
@@ -18,7 +18,7 @@
|
||||
/// to small JSONL events.
|
||||
///
|
||||
/// Storage layout:
|
||||
/// ```
|
||||
/// ```text
|
||||
/// {prefix}/
|
||||
/// batch_0001776319628000123.jsonl
|
||||
/// batch_0001776319745987654.jsonl
|
||||
@@ -221,15 +221,15 @@ pub struct CompactStats {
|
||||
pub new_key: Option<String>,
|
||||
}
|
||||
|
||||
// Proactively flush on drop, best-effort.
|
||||
// We can't `.await` in Drop; we spawn the flush on the tokio runtime if one
|
||||
// is available. If the runtime is already shutting down the flush is lost —
|
||||
// acceptable for these observability journals, which are hints not records.
|
||||
// Log unflushed-buffer size on drop. We can't `.await` from a sync `Drop`,
|
||||
// so a real flush isn't possible here — callers are responsible for calling
|
||||
// `.flush()` before dropping if durability matters. These journals are
|
||||
// observability hints; a few lost buffered events at shutdown are
|
||||
// acceptable per ADR-018.
|
||||
impl Drop for AppendLog {
|
||||
fn drop(&mut self) {
|
||||
let buf_len = self.buffer.try_lock().map(|b| b.len()).unwrap_or(0);
|
||||
if buf_len == 0 { return; }
|
||||
// Can't spawn from sync Drop on every runtime shape; log + move on.
|
||||
tracing::debug!(
|
||||
"append_log '{}' dropping with {} unflushed events",
|
||||
self.prefix, buf_len,
|
||||
@@ -237,3 +237,94 @@ impl Drop for AppendLog {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use object_store::memory::InMemory;
|
||||
|
||||
fn mk(threshold: usize) -> AppendLog {
|
||||
AppendLog::new(Arc::new(InMemory::new()), "prefix")
|
||||
.with_flush_threshold(threshold)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn append_stays_buffered_below_threshold() {
|
||||
let log = mk(5);
|
||||
log.append(b"one".to_vec()).await.unwrap();
|
||||
log.append(b"two".to_vec()).await.unwrap();
|
||||
assert_eq!(log.file_count().await.unwrap(), 0, "no files until threshold");
|
||||
let all = log.read_all().await.unwrap();
|
||||
assert_eq!(all, vec![b"one".to_vec(), b"two".to_vec()],
|
||||
"read_all surfaces unflushed buffer");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn append_auto_flushes_on_threshold() {
|
||||
let log = mk(3);
|
||||
for i in 0..3 {
|
||||
log.append(format!("evt{i}").into_bytes()).await.unwrap();
|
||||
}
|
||||
assert_eq!(log.file_count().await.unwrap(), 1, "threshold triggered one flush");
|
||||
|
||||
// A fourth append stays buffered until the next threshold.
|
||||
log.append(b"evt3".to_vec()).await.unwrap();
|
||||
assert_eq!(log.file_count().await.unwrap(), 1, "below threshold again");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn flush_empty_is_noop() {
|
||||
let log = mk(32);
|
||||
log.flush().await.unwrap();
|
||||
log.flush().await.unwrap();
|
||||
assert_eq!(log.file_count().await.unwrap(), 0);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn read_all_orders_events_across_flushes() {
|
||||
let log = mk(1); // flush-on-every-append
|
||||
for i in 0..5 {
|
||||
log.append(format!("e{i}").into_bytes()).await.unwrap();
|
||||
// Spread writes out so timestamps sort strictly.
|
||||
tokio::time::sleep(std::time::Duration::from_millis(2)).await;
|
||||
}
|
||||
let all = log.read_all().await.unwrap();
|
||||
let strs: Vec<String> = all.into_iter()
|
||||
.map(|v| String::from_utf8(v).unwrap())
|
||||
.collect();
|
||||
assert_eq!(strs, vec!["e0", "e1", "e2", "e3", "e4"],
|
||||
"lex sort of batch keys == chronological event order");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn compact_merges_multiple_files_into_one() {
|
||||
let log = mk(1); // force file-per-append
|
||||
for i in 0..4 {
|
||||
log.append(format!("e{i}").into_bytes()).await.unwrap();
|
||||
tokio::time::sleep(std::time::Duration::from_millis(2)).await;
|
||||
}
|
||||
assert_eq!(log.file_count().await.unwrap(), 4);
|
||||
|
||||
let stats = log.compact().await.unwrap();
|
||||
assert_eq!(stats.merged_files, 4);
|
||||
assert_eq!(stats.events, 4);
|
||||
assert!(stats.new_key.is_some());
|
||||
|
||||
assert_eq!(log.file_count().await.unwrap(), 1, "originals deleted, 1 survivor");
|
||||
let all = log.read_all().await.unwrap();
|
||||
assert_eq!(all.len(), 4, "no events lost in compaction");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn compact_with_single_file_is_noop() {
|
||||
let log = mk(1);
|
||||
log.append(b"only".to_vec()).await.unwrap();
|
||||
assert_eq!(log.file_count().await.unwrap(), 1);
|
||||
|
||||
let stats = log.compact().await.unwrap();
|
||||
assert_eq!(stats.merged_files, 1);
|
||||
assert_eq!(stats.events, 0, "nothing to consolidate");
|
||||
assert!(stats.new_key.is_none(), "no new file written");
|
||||
assert_eq!(log.file_count().await.unwrap(), 1, "original untouched");
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -198,3 +198,92 @@ impl ErrorJournal {
|
||||
self.log.file_count().await
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use object_store::memory::InMemory;
|
||||
|
||||
fn mk() -> ErrorJournal {
|
||||
ErrorJournal::new(Arc::new(InMemory::new()))
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn append_updates_ring_immediately() {
|
||||
let j = mk();
|
||||
j.append(BucketErrorEvent::new_read("tenant_a", "k1", "timeout")).await;
|
||||
j.append(BucketErrorEvent::new_write("tenant_b", "k2", "denied")).await;
|
||||
|
||||
let recent = j.recent(10).await;
|
||||
assert_eq!(recent.len(), 2);
|
||||
assert_eq!(recent[0].target, "tenant_a");
|
||||
assert_eq!(recent[1].target, "tenant_b");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn load_recent_rehydrates_ring_from_disk() {
|
||||
// Use a shared InMemory store so the second journal reads the first
|
||||
// journal's writes — mirrors gateway restart semantics.
|
||||
let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
|
||||
{
|
||||
let j1 = ErrorJournal {
|
||||
log: Arc::new(AppendLog::new(store.clone(), JOURNAL_PREFIX).with_flush_threshold(1)),
|
||||
ring: Arc::new(RwLock::new(VecDeque::with_capacity(RING_CAPACITY))),
|
||||
};
|
||||
j1.append(BucketErrorEvent::new_read("t", "k", "err")).await;
|
||||
j1.flush().await.unwrap();
|
||||
}
|
||||
|
||||
let j2 = ErrorJournal {
|
||||
log: Arc::new(AppendLog::new(store.clone(), JOURNAL_PREFIX)),
|
||||
ring: Arc::new(RwLock::new(VecDeque::with_capacity(RING_CAPACITY))),
|
||||
};
|
||||
assert_eq!(j2.recent(10).await.len(), 0, "fresh ring is empty");
|
||||
|
||||
let loaded = j2.load_recent().await.unwrap();
|
||||
assert_eq!(loaded, 1);
|
||||
assert_eq!(j2.recent(10).await[0].target, "t");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn mark_rescued_updates_most_recent_match() {
|
||||
let j = mk();
|
||||
j.append(BucketErrorEvent::new_read("t1", "k", "err")).await;
|
||||
j.append(BucketErrorEvent::new_read("t2", "k", "err")).await;
|
||||
j.append(BucketErrorEvent::new_read("t1", "k", "err")).await;
|
||||
|
||||
j.mark_rescued_last("t1", "k").await;
|
||||
|
||||
let recent = j.recent(10).await;
|
||||
assert!(!recent[0].rescued, "older t1 event not touched");
|
||||
assert!(!recent[1].rescued, "t2 not touched");
|
||||
assert!(recent[2].rescued, "most recent t1 was rescued");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn health_flags_buckets_with_three_or_more_errors() {
|
||||
let j = mk();
|
||||
for _ in 0..3 { j.append(BucketErrorEvent::new_read("broken", "k", "e")).await; }
|
||||
j.append(BucketErrorEvent::new_read("flaky", "k", "e")).await;
|
||||
|
||||
let h = j.health(60).await;
|
||||
assert_eq!(h.total_errors, 4);
|
||||
assert_eq!(h.per_bucket.get("broken"), Some(&3));
|
||||
assert_eq!(h.per_bucket.get("flaky"), Some(&1));
|
||||
assert!(h.unhealthy_buckets.contains(&"broken".to_string()));
|
||||
assert!(!h.unhealthy_buckets.contains(&"flaky".to_string()));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn filter_by_bucket_returns_chronological_order() {
|
||||
let j = mk();
|
||||
j.append(BucketErrorEvent::new_read("a", "k1", "e")).await;
|
||||
j.append(BucketErrorEvent::new_read("b", "k2", "e")).await;
|
||||
j.append(BucketErrorEvent::new_read("a", "k3", "e")).await;
|
||||
|
||||
let only_a = j.filter(Some("a"), None, 10).await;
|
||||
assert_eq!(only_a.len(), 2);
|
||||
assert_eq!(only_a[0].key, "k1", "oldest first");
|
||||
assert_eq!(only_a[1].key, "k3");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -205,3 +205,13 @@ tr:hover td { background: var(--accent-glow); }
|
||||
padding: 8px 12px; border-bottom: 1px solid var(--border); font-size: 13px;
|
||||
}
|
||||
.table-item:hover { background: var(--accent-glow); }
|
||||
|
||||
/* Phase 19 — Playbook panel */
|
||||
.boosted-row { background: rgba(120, 200, 120, 0.10); }
|
||||
.boosted-row td { border-top: 1px solid rgba(120, 200, 120, 0.30); }
|
||||
.mono-cell {
|
||||
font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
|
||||
font-size: 11px; color: var(--text-dim);
|
||||
max-width: 220px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
|
||||
}
|
||||
.panel-section + .panel-section { margin-top: 18px; }
|
||||
|
||||
+429
-32
@@ -178,9 +178,116 @@ enum Tab {
|
||||
Explore,
|
||||
Sql,
|
||||
Ingest,
|
||||
Playbook,
|
||||
Status,
|
||||
}
|
||||
|
||||
// --- Playbook memory types (Phase 19) ---
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, PartialEq)]
|
||||
struct PlaybookStats {
|
||||
entries: usize,
|
||||
entries_with_embeddings: usize,
|
||||
#[serde(default)]
|
||||
total_names_endorsed: usize,
|
||||
#[serde(default)]
|
||||
sample: Vec<PlaybookSample>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, PartialEq)]
|
||||
struct PlaybookSample {
|
||||
id: String,
|
||||
operation: String,
|
||||
#[serde(default)]
|
||||
city: Option<String>,
|
||||
#[serde(default)]
|
||||
state: Option<String>,
|
||||
#[serde(default)]
|
||||
endorsed: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, PartialEq)]
|
||||
struct HybridResp {
|
||||
#[serde(default)]
|
||||
sql_matches: usize,
|
||||
#[serde(default)]
|
||||
vector_reranked: usize,
|
||||
#[serde(default)]
|
||||
method: String,
|
||||
#[serde(default)]
|
||||
duration_ms: u64,
|
||||
#[serde(default)]
|
||||
answer: Option<String>,
|
||||
#[serde(default)]
|
||||
sources: Vec<HybridSource>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, PartialEq)]
|
||||
struct HybridSource {
|
||||
doc_id: String,
|
||||
chunk_text: String,
|
||||
score: f32,
|
||||
#[serde(default)]
|
||||
sql_verified: bool,
|
||||
#[serde(default)]
|
||||
playbook_boost: f32,
|
||||
#[serde(default)]
|
||||
playbook_citations: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, PartialEq)]
|
||||
struct IndexInfo {
|
||||
index_name: String,
|
||||
source: String,
|
||||
#[serde(default)]
|
||||
chunk_count: usize,
|
||||
#[serde(default)]
|
||||
vector_backend: String,
|
||||
}
|
||||
|
||||
async fn fetch_playbook_stats() -> Result<PlaybookStats, String> {
|
||||
let resp = reqwest::get(&format!("{}/vectors/playbook_memory/stats", api_base()))
|
||||
.await.map_err(|e| e.to_string())?;
|
||||
if !resp.status().is_success() {
|
||||
return Err(format!("HTTP {}: {}", resp.status(), resp.text().await.unwrap_or_default()));
|
||||
}
|
||||
resp.json().await.map_err(|e| e.to_string())
|
||||
}
|
||||
|
||||
async fn rebuild_playbook_memory() -> Result<serde_json::Value, String> {
|
||||
let client = reqwest::Client::new();
|
||||
let resp = client.post(&format!("{}/vectors/playbook_memory/rebuild", api_base()))
|
||||
.json(&serde_json::json!({}))
|
||||
.send().await.map_err(|e| e.to_string())?;
|
||||
if !resp.status().is_success() {
|
||||
return Err(format!("HTTP {}: {}", resp.status(), resp.text().await.unwrap_or_default()));
|
||||
}
|
||||
resp.json().await.map_err(|e| e.to_string())
|
||||
}
|
||||
|
||||
async fn fetch_indexes() -> Result<Vec<IndexInfo>, String> {
|
||||
let resp = reqwest::get(&format!("{}/vectors/indexes", api_base()))
|
||||
.await.map_err(|e| e.to_string())?;
|
||||
resp.json().await.map_err(|e| e.to_string())
|
||||
}
|
||||
|
||||
async fn hybrid_search(index_name: &str, question: &str, use_playbook: bool, top_k: usize) -> Result<HybridResp, String> {
|
||||
let client = reqwest::Client::new();
|
||||
let resp = client.post(&format!("{}/vectors/hybrid", api_base()))
|
||||
.json(&serde_json::json!({
|
||||
"index_name": index_name,
|
||||
"question": question,
|
||||
"top_k": top_k,
|
||||
"generate": false,
|
||||
"use_playbook_memory": use_playbook,
|
||||
}))
|
||||
.send().await.map_err(|e| e.to_string())?;
|
||||
if !resp.status().is_success() {
|
||||
return Err(format!("HTTP {}: {}", resp.status(), resp.text().await.unwrap_or_default()));
|
||||
}
|
||||
resp.json().await.map_err(|e| e.to_string())
|
||||
}
|
||||
|
||||
// --- App ---
|
||||
|
||||
#[component]
|
||||
@@ -239,6 +346,11 @@ fn App() -> Element {
|
||||
onclick: move |_| active_tab.set(Tab::Ingest),
|
||||
"Ingest"
|
||||
}
|
||||
button {
|
||||
class: if *active_tab.read() == Tab::Playbook { "tab active" } else { "tab" },
|
||||
onclick: move |_| active_tab.set(Tab::Playbook),
|
||||
"Playbook"
|
||||
}
|
||||
button {
|
||||
class: if *active_tab.read() == Tab::Status { "tab active" } else { "tab" },
|
||||
onclick: move |_| active_tab.set(Tab::Status),
|
||||
@@ -260,6 +372,7 @@ fn App() -> Element {
|
||||
Tab::Explore => rsx! { ExplorePanel { datasets: datasets.read().clone() } },
|
||||
Tab::Sql => rsx! { SqlPanel {} },
|
||||
Tab::Ingest => rsx! { IngestPanel {} },
|
||||
Tab::Playbook => rsx! { PlaybookPanel {} },
|
||||
Tab::Status => rsx! { StatusPanel {} },
|
||||
}
|
||||
}
|
||||
@@ -354,14 +467,14 @@ fn AskPanel(datasets: Vec<Dataset>) -> Element {
|
||||
div { class: "panel ask-panel",
|
||||
div { class: "ask-hero",
|
||||
h2 { "Ask your data anything" }
|
||||
p { class: "subtitle", "Natural language → SQL → Results. Powered by local AI." }
|
||||
p { class: "subtitle", "Natural language → SQL → Results. Powered by local AI over the staffing dataset." }
|
||||
}
|
||||
|
||||
div { class: "ask-input-row",
|
||||
input {
|
||||
class: "ask-input",
|
||||
value: "{question}",
|
||||
placeholder: "e.g. Which department has the highest average salary?",
|
||||
placeholder: "e.g. Which clients placed the most candidates last quarter?",
|
||||
oninput: move |e| question.set(e.value()),
|
||||
onkeydown: move |e| {
|
||||
if e.key() == Key::Enter {
|
||||
@@ -432,10 +545,12 @@ fn AskPanel(datasets: Vec<Dataset>) -> Element {
|
||||
|
||||
div { class: "ask-examples",
|
||||
"Try: "
|
||||
button { class: "example-btn", onclick: move |_| question.set("Which department has the highest average salary?".into()), "highest avg salary by dept" }
|
||||
button { class: "example-btn", onclick: move |_| question.set("Show me the top 3 most expensive products".into()), "top 3 expensive products" }
|
||||
button { class: "example-btn", onclick: move |_| question.set("How many events per action type?".into()), "events by action" }
|
||||
button { class: "example-btn", onclick: move |_| question.set("List all employees who earn more than 90000".into()), "employees > 90k" }
|
||||
button { class: "example-btn", onclick: move |_| question.set("How many candidates do we have by city?".into()), "candidates by city" }
|
||||
button { class: "example-btn", onclick: move |_| question.set("Top 10 clients by total placements".into()), "top clients by placements" }
|
||||
button { class: "example-btn", onclick: move |_| question.set("Open job orders ordered by bill rate descending".into()), "open jobs by rate" }
|
||||
button { class: "example-btn", onclick: move |_| question.set("Recruiters with the highest placement count".into()), "top recruiters" }
|
||||
button { class: "example-btn", onclick: move |_| question.set("Total billed hours per client last month".into()), "hours per client" }
|
||||
button { class: "example-btn", onclick: move |_| question.set("Cold leads: candidates we called more than 5 times but never placed".into()), "cold leads" }
|
||||
}
|
||||
|
||||
if let Some(sql) = generated_sql.read().as_ref() {
|
||||
@@ -578,7 +693,7 @@ fn ExplorePanel(datasets: Vec<Dataset>) -> Element {
|
||||
|
||||
#[component]
|
||||
fn SqlPanel() -> Element {
|
||||
let mut query_text = use_signal(|| String::from("SELECT * FROM employees LIMIT 10"));
|
||||
let mut query_text = use_signal(|| String::from("SELECT candidate_id, first_name, last_name, city, status FROM candidates LIMIT 10"));
|
||||
let mut result = use_signal(|| None::<Result<QueryResponse, String>>);
|
||||
let mut loading = use_signal(|| false);
|
||||
|
||||
@@ -727,7 +842,7 @@ fn DashboardPanel() -> Element {
|
||||
}
|
||||
div { class: "stat-card accent",
|
||||
div { class: "stat-value", "{s[\"hnsw_loaded\"]}" }
|
||||
div { class: "stat-label", "HNSW Indexes (27ms)" }
|
||||
div { class: "stat-label", "HNSW Indexes Loaded" }
|
||||
}
|
||||
div { class: "stat-card",
|
||||
div { class: "stat-value", "{s[\"tools\"]}" }
|
||||
@@ -750,27 +865,27 @@ fn DashboardPanel() -> Element {
|
||||
div { class: "arch-grid",
|
||||
div { class: "arch-card",
|
||||
div { class: "arch-title", "Ingest" }
|
||||
div { class: "arch-items", "CSV, JSON, PDF, Text, PostgreSQL, File Watcher" }
|
||||
div { class: "arch-items", "CSV · JSON · PDF (+OCR) · Text · Postgres · MySQL · Inbox watcher · Cron schedules" }
|
||||
}
|
||||
div { class: "arch-card",
|
||||
div { class: "arch-title", "Storage" }
|
||||
div { class: "arch-items", "Parquet on Object Storage, Delta Writes, Compaction" }
|
||||
div { class: "arch-items", "Parquet on Object Storage · Delta writes · Compaction · Tombstones · Multi-bucket federation + rescue" }
|
||||
}
|
||||
div { class: "arch-card",
|
||||
div { class: "arch-title", "Query" }
|
||||
div { class: "arch-items", "DataFusion SQL, MemCache (9.8x), Hot/Cold" }
|
||||
div { class: "arch-items", "DataFusion SQL · MemCache (9.8× hot) · Merge-on-read · AI-safe views" }
|
||||
}
|
||||
div { class: "arch-card",
|
||||
div { class: "arch-title", "AI" }
|
||||
div { class: "arch-items", "Ollama (local), Embed, Generate, RAG, HNSW" }
|
||||
div { class: "arch-title", "AI / Vector" }
|
||||
div { class: "arch-items", "Ollama (local) · Embed/Generate/RAG · HNSW (Parquet) · Lance IVF_PQ · Hybrid SQL+vector · Profile-scoped" }
|
||||
}
|
||||
div { class: "arch-card",
|
||||
div { class: "arch-title", "Learning loop" }
|
||||
div { class: "arch-items", "Playbook memory · Endorsement boost · Multi-agent orchestrator · Autotune agent (Pareto-promote)" }
|
||||
}
|
||||
div { class: "arch-card",
|
||||
div { class: "arch-title", "Governance" }
|
||||
div { class: "arch-items", "Event Journal, PII Detection, Tool Registry, Access Control" }
|
||||
}
|
||||
div { class: "arch-card",
|
||||
div { class: "arch-title", "Agents" }
|
||||
div { class: "arch-items", "Workspaces, Handoff, Shortlists, Activity Logs" }
|
||||
div { class: "arch-items", "Event journal · PII detection · Tool registry · Access control · Audit log · Catalog v2 metadata" }
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -779,20 +894,23 @@ fn DashboardPanel() -> Element {
|
||||
h3 { "Build Progression" }
|
||||
div { class: "phase-list",
|
||||
{rsx! {
|
||||
PhaseItem { num: "0-5", name: "Foundation", detail: "Storage, Catalog, DataFusion, AI, UI, gRPC" }
|
||||
PhaseItem { num: "6", name: "Ingest Pipeline", detail: "CSV/JSON/PDF/Text auto-schema" }
|
||||
PhaseItem { num: "7", name: "Vector + RAG", detail: "Embed, Search, LLM Answers" }
|
||||
PhaseItem { num: "8", name: "Hot Cache", detail: "9.8x speedup, Delta Writes" }
|
||||
PhaseItem { num: "8.5", name: "Agent Workspaces", detail: "Per-contract, Instant Handoff" }
|
||||
PhaseItem { num: "9", name: "Event Journal", detail: "Append-only Mutation History" }
|
||||
PhaseItem { num: "10", name: "Rich Catalog", detail: "PII Detection, Lineage" }
|
||||
PhaseItem { num: "11", name: "Embedding Versioning", detail: "Model-proof Vectors" }
|
||||
PhaseItem { num: "12", name: "Tool Registry", detail: "6 Governed Actions + Audit" }
|
||||
PhaseItem { num: "13", name: "Access Control", detail: "Role-based, Field-level" }
|
||||
PhaseItem { num: "14", name: "Schema Evolution", detail: "Diff Detection, AI Migration" }
|
||||
PhaseItem { num: "15", name: "HNSW Index", detail: "100K Search in 27ms" }
|
||||
PhaseItem { num: "16", name: "File Watcher", detail: "Auto-ingest from Inbox" }
|
||||
PhaseItem { num: "17", name: "DB Connector", detail: "PostgreSQL Import" }
|
||||
PhaseItem { num: "0-5", name: "Foundation", detail: "Storage · Catalog · DataFusion · Ollama · UI · gRPC" }
|
||||
PhaseItem { num: "6", name: "Ingest Pipeline", detail: "CSV · JSON · PDF · Text · auto-schema · dedupe" }
|
||||
PhaseItem { num: "7", name: "Vector + RAG", detail: "Embed · brute-force cosine · LLM grounded answers" }
|
||||
PhaseItem { num: "8", name: "Hot Cache + Deltas", detail: "MemTable LRU · 9.8× speedup · merge-on-read · compaction" }
|
||||
PhaseItem { num: "8.5", name: "Agent Workspaces", detail: "Per-contract · daily/weekly/monthly tiers · zero-copy handoff" }
|
||||
PhaseItem { num: "9", name: "Event Journal", detail: "Append-only mutation log · time-travel · audit" }
|
||||
PhaseItem { num: "10", name: "Rich Catalog v2", detail: "PII auto-detection · lineage · freshness SLA · sensitivity" }
|
||||
PhaseItem { num: "11", name: "Embedding Versioning", detail: "Per-index model+version · A/B · incremental re-embed" }
|
||||
PhaseItem { num: "12", name: "Tool Registry", detail: "Governed actions · param validation · audit · MCP-ready" }
|
||||
PhaseItem { num: "13", name: "Access Control", detail: "Roles · field-level sensitivity · column masking · query audit" }
|
||||
PhaseItem { num: "14", name: "Schema Evolution", detail: "Diff detection · AI migration prompts · versioned schemas" }
|
||||
PhaseItem { num: "15", name: "HNSW + Trials", detail: "100K vectors · p50 873µs · trial journal · eval harness" }
|
||||
PhaseItem { num: "16", name: "Hot-swap + Autotune", detail: "Promotion registry · rollback · ε-greedy agent · Pareto winner" }
|
||||
PhaseItem { num: "17", name: "Model Profiles + VRAM", detail: "ModelProfile manifests · scoped search · sequential model swap" }
|
||||
PhaseItem { num: "18", name: "Lance hybrid backend", detail: "IVF_PQ build 14× faster · random fetch 112× · S3-native · per-profile routing" }
|
||||
PhaseItem { num: "19", name: "Playbook memory", detail: "Feedback loop · endorsement boost (cap 0.25) · orchestrator write-through · citations" }
|
||||
PhaseItem { num: "+", name: "Federation + Schedules", detail: "Multi-bucket · rescue fallback · error journal · MySQL · PDF OCR · cron ingest · catalog dedupe" }
|
||||
}}
|
||||
}
|
||||
}
|
||||
@@ -816,6 +934,285 @@ fn PhaseItem(num: String, name: String, detail: String) -> Element {
|
||||
}
|
||||
}
|
||||
|
||||
// === PLAYBOOK — Phase 19 meta-index feedback loop ===
|
||||
|
||||
#[component]
|
||||
fn PlaybookPanel() -> Element {
|
||||
let mut stats = use_signal(|| None::<Result<PlaybookStats, String>>);
|
||||
let mut indexes = use_signal(Vec::<IndexInfo>::new);
|
||||
let mut rebuild_status = use_signal(|| None::<Result<String, String>>);
|
||||
let mut rebuilding = use_signal(|| false);
|
||||
let mut loaded = use_signal(|| false);
|
||||
|
||||
// Comparison state
|
||||
let mut selected_index = use_signal(|| String::new());
|
||||
let mut question = use_signal(|| String::from("reliable assembler in Detroit"));
|
||||
let mut top_k = use_signal(|| 10usize);
|
||||
let mut compare_loading = use_signal(|| false);
|
||||
let mut hits_off = use_signal(|| None::<Result<HybridResp, String>>);
|
||||
let mut hits_on = use_signal(|| None::<Result<HybridResp, String>>);
|
||||
|
||||
let load_all = move || {
|
||||
spawn(async move {
|
||||
stats.set(Some(fetch_playbook_stats().await));
|
||||
if let Ok(ix) = fetch_indexes().await {
|
||||
if selected_index.read().is_empty() {
|
||||
if let Some(default) = ix.iter().find(|i| i.source == "workers_500k").or_else(|| ix.first()) {
|
||||
selected_index.set(default.index_name.clone());
|
||||
}
|
||||
}
|
||||
indexes.set(ix);
|
||||
}
|
||||
});
|
||||
};
|
||||
|
||||
use_effect(move || {
|
||||
if !*loaded.read() {
|
||||
loaded.set(true);
|
||||
load_all();
|
||||
}
|
||||
});
|
||||
|
||||
let do_rebuild = move |_| {
|
||||
spawn(async move {
|
||||
rebuilding.set(true);
|
||||
rebuild_status.set(None);
|
||||
match rebuild_playbook_memory().await {
|
||||
Ok(v) => rebuild_status.set(Some(Ok(format!("rebuild ok — {}", v)))),
|
||||
Err(e) => rebuild_status.set(Some(Err(e))),
|
||||
}
|
||||
// Refresh stats afterward
|
||||
stats.set(Some(fetch_playbook_stats().await));
|
||||
rebuilding.set(false);
|
||||
});
|
||||
};
|
||||
|
||||
let do_compare = move |_| {
|
||||
let idx = selected_index.read().clone();
|
||||
let q = question.read().clone();
|
||||
let k = *top_k.read();
|
||||
if idx.is_empty() || q.trim().is_empty() { return; }
|
||||
spawn(async move {
|
||||
compare_loading.set(true);
|
||||
hits_off.set(None);
|
||||
hits_on.set(None);
|
||||
// Run both sequentially so the embedding cache is shared
|
||||
hits_off.set(Some(hybrid_search(&idx, &q, false, k).await));
|
||||
hits_on.set(Some(hybrid_search(&idx, &q, true, k).await));
|
||||
compare_loading.set(false);
|
||||
});
|
||||
};
|
||||
|
||||
rsx! {
|
||||
div { class: "panel",
|
||||
div { class: "ask-hero",
|
||||
h2 { "Playbook Memory" }
|
||||
p { class: "subtitle",
|
||||
"Phase 19 feedback loop: past successful playbooks boost future search rankings. \
|
||||
Endorsed workers from semantically similar past operations re-rank toward the top, \
|
||||
with citations back to the playbook that endorsed them."
|
||||
}
|
||||
}
|
||||
|
||||
// Stats card
|
||||
div { class: "panel-section",
|
||||
match stats.read().as_ref() {
|
||||
None => rsx! { div { class: "loading", "loading playbook stats..." } },
|
||||
Some(Err(e)) => rsx! { div { class: "error", "stats: {e}" } },
|
||||
Some(Ok(s)) => rsx! {
|
||||
div { class: "stat-grid",
|
||||
div { class: "stat-card",
|
||||
div { class: "stat-value", "{s.entries}" }
|
||||
div { class: "stat-label", "Playbooks in Memory" }
|
||||
}
|
||||
div { class: "stat-card",
|
||||
div { class: "stat-value", "{s.entries_with_embeddings}" }
|
||||
div { class: "stat-label", "Embedded" }
|
||||
}
|
||||
div { class: "stat-card accent",
|
||||
div { class: "stat-value", "{s.total_names_endorsed}" }
|
||||
div { class: "stat-label", "Endorsed Worker-Tags" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
div { class: "sql-actions",
|
||||
button {
|
||||
class: "btn",
|
||||
disabled: *rebuilding.read(),
|
||||
onclick: do_rebuild,
|
||||
if *rebuilding.read() { "rebuilding from successful_playbooks..." } else { "Rebuild from successful_playbooks" }
|
||||
}
|
||||
}
|
||||
if let Some(s) = rebuild_status.read().as_ref() {
|
||||
match s {
|
||||
Ok(msg) => rsx! { div { class: "result-box", "{msg}" } },
|
||||
Err(e) => rsx! { div { class: "error", "{e}" } },
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Sample playbooks
|
||||
if let Some(Ok(s)) = stats.read().as_ref() {
|
||||
if !s.sample.is_empty() {
|
||||
div { class: "panel-section",
|
||||
h3 { "Sample playbooks" }
|
||||
div { class: "table-wrap",
|
||||
table {
|
||||
thead { tr {
|
||||
th { "ID" }
|
||||
th { "Operation" }
|
||||
th { "Location" }
|
||||
th { "Endorsed" }
|
||||
} }
|
||||
tbody {
|
||||
for pb in s.sample.iter() {
|
||||
{
|
||||
let loc = match (&pb.city, &pb.state) {
|
||||
(Some(c), Some(st)) => format!("{c}, {st}"),
|
||||
_ => "—".into(),
|
||||
};
|
||||
let endorsed = if pb.endorsed.is_empty() {
|
||||
"—".to_string()
|
||||
} else {
|
||||
pb.endorsed.join(", ")
|
||||
};
|
||||
let pid = pb.id.clone();
|
||||
let op = pb.operation.clone();
|
||||
rsx! {
|
||||
tr {
|
||||
td { class: "mono-cell", title: "{pid}", "{pid}" }
|
||||
td { "{op}" }
|
||||
td { "{loc}" }
|
||||
td { "{endorsed}" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Side-by-side comparison: boost OFF vs ON
|
||||
div { class: "panel-section",
|
||||
h3 { "See the boost — search compared" }
|
||||
p { class: "hint",
|
||||
"Run the same query against the same index twice — once with playbook boost OFF and once ON. \
|
||||
Hits with non-zero playbook_boost and citations are workers that past similar playbooks endorsed."
|
||||
}
|
||||
div { class: "form-row",
|
||||
label { "Index" }
|
||||
select {
|
||||
value: "{selected_index}",
|
||||
onchange: move |e| selected_index.set(e.value()),
|
||||
for ix in indexes.read().iter() {
|
||||
option { value: "{ix.index_name}", "{ix.index_name} ({ix.source}, {ix.chunk_count} chunks, {ix.vector_backend})" }
|
||||
}
|
||||
}
|
||||
}
|
||||
div { class: "form-row",
|
||||
label { "Question" }
|
||||
input {
|
||||
value: "{question}",
|
||||
oninput: move |e| question.set(e.value()),
|
||||
placeholder: "e.g. reliable assembler in Detroit"
|
||||
}
|
||||
}
|
||||
div { class: "form-row",
|
||||
label { "Top K" }
|
||||
input {
|
||||
r#type: "number",
|
||||
value: "{top_k}",
|
||||
oninput: move |e| {
|
||||
if let Ok(n) = e.value().parse::<usize>() { top_k.set(n.clamp(1, 50)); }
|
||||
}
|
||||
}
|
||||
}
|
||||
button {
|
||||
class: "btn btn-ask",
|
||||
disabled: *compare_loading.read(),
|
||||
onclick: do_compare,
|
||||
if *compare_loading.read() { "running both queries..." } else { "Run comparison" }
|
||||
}
|
||||
|
||||
div { class: "explore-grid",
|
||||
div { class: "ds-detail",
|
||||
h3 { "Boost OFF (vanilla)" }
|
||||
match hits_off.read().as_ref() {
|
||||
None => rsx! { div { class: "empty", "—" } },
|
||||
Some(Err(e)) => rsx! { div { class: "error", "{e}" } },
|
||||
Some(Ok(r)) => rsx! { HybridHitTable { resp: r.clone() } },
|
||||
}
|
||||
}
|
||||
div { class: "ds-detail",
|
||||
h3 { "Boost ON (Phase 19)" }
|
||||
match hits_on.read().as_ref() {
|
||||
None => rsx! { div { class: "empty", "—" } },
|
||||
Some(Err(e)) => rsx! { div { class: "error", "{e}" } },
|
||||
Some(Ok(r)) => rsx! { HybridHitTable { resp: r.clone() } },
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[component]
|
||||
fn HybridHitTable(resp: HybridResp) -> Element {
|
||||
rsx! {
|
||||
div { class: "results-info",
|
||||
"{resp.sources.len()} hits · {resp.duration_ms}ms · method={resp.method}"
|
||||
}
|
||||
if resp.sources.is_empty() {
|
||||
div { class: "empty-sm", "no hits" }
|
||||
} else {
|
||||
div { class: "table-wrap",
|
||||
table {
|
||||
thead { tr {
|
||||
th { "#" }
|
||||
th { "Doc" }
|
||||
th { "Score" }
|
||||
th { "Boost" }
|
||||
th { "Citations" }
|
||||
th { "Snippet" }
|
||||
} }
|
||||
tbody {
|
||||
for (i, h) in resp.sources.iter().enumerate() {
|
||||
{
|
||||
let snippet: String = h.chunk_text.chars().take(120).collect();
|
||||
let cites = if h.playbook_citations.is_empty() {
|
||||
"—".to_string()
|
||||
} else {
|
||||
h.playbook_citations.join(", ")
|
||||
};
|
||||
let row_class = if h.playbook_boost > 0.0 { "boosted-row" } else { "" };
|
||||
let rank = i + 1;
|
||||
let did = h.doc_id.clone();
|
||||
let score = format!("{:.3}", h.score);
|
||||
let boost = if h.playbook_boost > 0.0 { format!("+{:.3}", h.playbook_boost) } else { "—".into() };
|
||||
rsx! {
|
||||
tr { class: "{row_class}",
|
||||
td { "{rank}" }
|
||||
td { class: "mono-cell", "{did}" }
|
||||
td { "{score}" }
|
||||
td { "{boost}" }
|
||||
td { class: "mono-cell", title: "{cites}", "{cites}" }
|
||||
td { "{snippet}" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// === INGEST — Data on-ramp ===
|
||||
|
||||
#[component]
|
||||
|
||||
@@ -357,7 +357,25 @@ impl LanceVectorStore {
|
||||
/// Search for top_k nearest neighbors of `query`. Uses the IVF_PQ
|
||||
/// index if one exists; otherwise does a full scan (slow but
|
||||
/// correct — useful during development before index build).
|
||||
pub async fn search(&self, query: &[f32], top_k: usize) -> Result<Vec<Hit>, String> {
|
||||
///
|
||||
/// `nprobes` tells Lance how many IVF partitions to probe per query.
|
||||
/// Lance's built-in default is **1**, which caps recall well below
|
||||
/// what the index is actually capable of. Passing `None` keeps that
|
||||
/// default (only sensible when latency trumps recall); for real
|
||||
/// workloads set `nprobes` to 5–10% of `num_partitions` (e.g. 20–30
|
||||
/// on a 316-partition index).
|
||||
///
|
||||
/// `refine_factor` re-ranks the PQ-approximate top-k by computing
|
||||
/// exact distances on `top_k * refine_factor` candidates, then
|
||||
/// trimming to `top_k`. Cheap way to buy back recall lost to product
|
||||
/// quantization. `None` skips the re-rank pass.
|
||||
pub async fn search(
|
||||
&self,
|
||||
query: &[f32],
|
||||
top_k: usize,
|
||||
nprobes: Option<usize>,
|
||||
refine_factor: Option<u32>,
|
||||
) -> Result<Vec<Hit>, String> {
|
||||
use lance::dataset::Dataset;
|
||||
|
||||
let dataset = Dataset::open(&self.path).await.map_err(e)?;
|
||||
@@ -365,6 +383,8 @@ impl LanceVectorStore {
|
||||
|
||||
let mut scanner = dataset.scan();
|
||||
scanner.nearest("vector", &qarr, top_k as usize).map_err(e)?;
|
||||
if let Some(n) = nprobes { scanner.nprobes(n); }
|
||||
if let Some(f) = refine_factor { scanner.refine(f); }
|
||||
scanner.project(&["doc_id", "chunk_text"]).map_err(e)?;
|
||||
|
||||
let mut stream = scanner.try_into_stream().await.map_err(e)?;
|
||||
|
||||
@@ -8,6 +8,8 @@ shared = { path = "../shared" }
|
||||
storaged = { path = "../storaged" }
|
||||
aibridge = { path = "../aibridge" }
|
||||
catalogd = { path = "../catalogd" }
|
||||
queryd = { path = "../queryd" }
|
||||
futures = { workspace = true }
|
||||
# ADR-019 firewall — vectord-lance owns its own Arrow 57 / Lance 4 deps.
|
||||
# Public API uses only std types so no version conflict propagates here.
|
||||
vectord-lance = { path = "../vectord-lance" }
|
||||
@@ -23,3 +25,4 @@ arrow = { workspace = true }
|
||||
chrono = { workspace = true }
|
||||
instant-distance = { workspace = true }
|
||||
uuid = { workspace = true }
|
||||
sha2 = { workspace = true }
|
||||
|
||||
+118
-40
@@ -204,6 +204,7 @@ pub struct AgentDeps {
|
||||
pub embedding_cache: EmbeddingCache,
|
||||
pub trial_journal: TrialJournal,
|
||||
pub promotion_registry: PromotionRegistry,
|
||||
pub harness_store: crate::harness::HarnessStore,
|
||||
}
|
||||
|
||||
// -------- Spawn --------
|
||||
@@ -354,12 +355,11 @@ async fn run_one_cycle(
|
||||
// Read history.
|
||||
let history = deps.trial_journal.list(&event.index_name).await
|
||||
.map_err(|e| format!("read journal: {e}"))?;
|
||||
if history.is_empty() {
|
||||
return Err(format!(
|
||||
"no trials yet for '{}' — seed with at least one POST /hnsw/trial first",
|
||||
event.index_name,
|
||||
));
|
||||
}
|
||||
|
||||
// Bootstrap mode: first ever visit to this index. Auto-generate a
|
||||
// harness (if missing) and seed with the default config — the Phase 15
|
||||
// known-good ec=80/es=30. Subsequent visits go through the proposer.
|
||||
let is_bootstrap = history.is_empty();
|
||||
|
||||
// Current champion (if any) is the promoted config.
|
||||
let champion = deps.promotion_registry.get_current(&event.index_name).await;
|
||||
@@ -367,24 +367,26 @@ async fn run_one_cycle(
|
||||
history.iter().find(|t| t.id == p.trial_id).cloned()
|
||||
});
|
||||
|
||||
// Propose the next config.
|
||||
let Some(next_config) = propose_next_config(&history, champion_trial.as_ref()) else {
|
||||
return Err("proposer returned None — search space exhausted".into());
|
||||
let (next_config, harness_name) = if is_bootstrap {
|
||||
let name = ensure_auto_harness(&event.index_name, deps).await?;
|
||||
(HnswConfig::default(), name)
|
||||
} else {
|
||||
let Some(cfg) = propose_next_config(&history, champion_trial.as_ref()) else {
|
||||
return Err("proposer returned None — search space exhausted".into());
|
||||
};
|
||||
// Validate bounds defensively.
|
||||
if !(10..=400).contains(&cfg.ef_construction) {
|
||||
return Err(format!("proposed ef_construction={} out of bounds", cfg.ef_construction));
|
||||
}
|
||||
if !(10..=200).contains(&cfg.ef_search) {
|
||||
return Err(format!("proposed ef_search={} out of bounds", cfg.ef_search));
|
||||
}
|
||||
// A future refinement: remember per-index "canonical harness" on
|
||||
// the index metadata. For now: latest wins.
|
||||
let hname = history.last().unwrap().eval_set.clone();
|
||||
(cfg, hname)
|
||||
};
|
||||
|
||||
// Validate bounds defensively.
|
||||
if !(10..=400).contains(&next_config.ef_construction) {
|
||||
return Err(format!("proposed ef_construction={} out of bounds", next_config.ef_construction));
|
||||
}
|
||||
if !(10..=200).contains(&next_config.ef_search) {
|
||||
return Err(format!("proposed ef_search={} out of bounds", next_config.ef_search));
|
||||
}
|
||||
|
||||
// Need a harness to measure. Use the most recent one in history.
|
||||
// (A future refinement: remember per-index "canonical harness" on
|
||||
// the index metadata. For now: latest wins.)
|
||||
let harness_name = history.last().unwrap().eval_set.clone();
|
||||
let mut harness_set = harness::EvalSet::load(&deps.store, &harness_name).await
|
||||
let mut harness_set = deps.harness_store.load_for_index(&event.index_name, &harness_name).await
|
||||
.map_err(|e| format!("load harness '{harness_name}': {e}"))?;
|
||||
|
||||
let embeddings = deps.embedding_cache.get_or_load(&event.index_name).await
|
||||
@@ -393,7 +395,7 @@ async fn run_one_cycle(
|
||||
if !harness_set.ground_truth_built {
|
||||
harness::compute_ground_truth(&mut harness_set, &embeddings, &deps.ai_client).await
|
||||
.map_err(|e| format!("ground truth: {e}"))?;
|
||||
harness_set.save(&deps.store).await.ok();
|
||||
deps.harness_store.save(&harness_set).await.ok();
|
||||
}
|
||||
|
||||
// Build + bench.
|
||||
@@ -428,10 +430,13 @@ async fn run_one_cycle(
|
||||
let dims = embeddings.first().map(|e| e.vector.len()).unwrap_or(0);
|
||||
let memory_bytes = (embeddings.len() * dims * std::mem::size_of::<f32>() + embeddings.len() * 128) as u64;
|
||||
|
||||
let note = match &event.reason {
|
||||
TriggerReason::DatasetAppended { dataset } => format!("agent: dataset_appended({dataset})"),
|
||||
TriggerReason::Manual => "agent: manual".to_string(),
|
||||
TriggerReason::Periodic => "agent: periodic".to_string(),
|
||||
let note = {
|
||||
let base = match &event.reason {
|
||||
TriggerReason::DatasetAppended { dataset } => format!("agent: dataset_appended({dataset})"),
|
||||
TriggerReason::Manual => "agent: manual".to_string(),
|
||||
TriggerReason::Periodic => "agent: periodic".to_string(),
|
||||
};
|
||||
if is_bootstrap { format!("{base} bootstrap") } else { base }
|
||||
};
|
||||
|
||||
let trial = Trial {
|
||||
@@ -597,19 +602,92 @@ pub fn propose_next_config(history: &[Trial], champion: Option<&Trial>) -> Optio
|
||||
|
||||
// -------- Helpers --------
|
||||
|
||||
/// Find an index to poke on a periodic wake. Strategy: the one with the
|
||||
/// most recent trial. If nothing's been trialed yet, return None.
|
||||
/// Minimum vectors for an index to be worth auto-tuning. Below this,
|
||||
/// HNSW's win over brute-force is too small to matter and trial budget
|
||||
/// is better spent on bigger indexes.
|
||||
const AUTOTUNE_MIN_VECTORS: usize = 1_000;
|
||||
|
||||
/// Find an index to poke on a periodic wake. Strategy: least-recently-tuned
|
||||
/// wins — pick the index (from `IndexRegistry`, not just promoted ones)
|
||||
/// whose most-recent trial is oldest, so trial budget spreads across the
|
||||
/// whole portfolio. Never-trialed indexes sort first (None < Some), which
|
||||
/// is exactly what we want: bootstrap them on their first visit.
|
||||
///
|
||||
/// Why not "most recently promoted" (the original strategy): a converged
|
||||
/// index like `threat_intel_v1` sits at recall=1.0 on a minimal graph and
|
||||
/// can't be improved — but it was always the freshest promotion, so the
|
||||
/// agent burned every trial on it while 40+ other indexes got zero
|
||||
/// attention. Rotating by last-trial-time fixes that without any explicit
|
||||
/// convergence detection.
|
||||
///
|
||||
/// Why IndexRegistry instead of PromotionRegistry: on a fresh system only
|
||||
/// a handful of indexes have ever been promoted, so promotion-based
|
||||
/// picking starves new ones. IndexRegistry lists every live index the
|
||||
/// platform knows about; filtering by `AUTOTUNE_MIN_VECTORS` keeps the
|
||||
/// proposer off indexes where the graph config doesn't matter.
|
||||
///
|
||||
/// Cost: N extra journal reads per periodic tick, where N = eligible
|
||||
/// indexes. Fine at dozens; if the portfolio grows into thousands, cache
|
||||
/// `last_trial_at` on the IndexMeta.
|
||||
async fn pick_periodic_target(deps: &AgentDeps) -> Option<String> {
|
||||
// `/agent` runs against any index that has a trial journal. We don't
|
||||
// have a "list all journals" helper, so we derive candidates from the
|
||||
// promotion registry (indexes with a human ever promoting are live).
|
||||
let promos = deps.promotion_registry.list_all().await.ok()?;
|
||||
// Prefer the one most recently promoted — it's the one a human cares
|
||||
// about right now.
|
||||
promos.into_iter()
|
||||
.filter_map(|f| f.current.map(|c| (f.index_name, c.promoted_at)))
|
||||
.max_by_key(|(_, at)| *at)
|
||||
.map(|(name, _)| name)
|
||||
let candidates: Vec<String> = deps
|
||||
.index_registry
|
||||
.list(None, None)
|
||||
.await
|
||||
.into_iter()
|
||||
.filter(|m| m.chunk_count >= AUTOTUNE_MIN_VECTORS)
|
||||
.map(|m| m.index_name)
|
||||
.collect();
|
||||
if candidates.is_empty() { return None; }
|
||||
|
||||
let mut scored: Vec<(String, Option<DateTime<Utc>>)> = Vec::with_capacity(candidates.len());
|
||||
for name in candidates {
|
||||
let last_trial_at = deps
|
||||
.trial_journal
|
||||
.list(&name)
|
||||
.await
|
||||
.ok()
|
||||
.and_then(|trials| trials.into_iter().map(|t| t.created_at).max());
|
||||
scored.push((name, last_trial_at));
|
||||
}
|
||||
|
||||
scored.into_iter().min_by_key(|(_, last)| *last).map(|(n, _)| n)
|
||||
}
|
||||
|
||||
/// On the first visit to an index, load or synthesize an eval harness so
|
||||
/// bootstrap trials have something to measure recall against. Returns the
|
||||
/// harness name. Synthetic harnesses sample 20 chunks and use their first
|
||||
/// 200 chars as self-queries — the ground truth is computed by brute-force
|
||||
/// cosine, so recall numbers are real (a config that finds the chunk in
|
||||
/// top-k counts as a hit).
|
||||
async fn ensure_auto_harness(index_name: &str, deps: &AgentDeps) -> Result<String, String> {
|
||||
let name = format!("{index_name}_auto");
|
||||
|
||||
if deps.harness_store.load_for_index(index_name, &name).await.is_ok() {
|
||||
return Ok(name);
|
||||
}
|
||||
|
||||
let embeddings = deps
|
||||
.embedding_cache
|
||||
.get_or_load(index_name)
|
||||
.await
|
||||
.map_err(|e| format!("load embeddings for auto-harness: {e}"))?;
|
||||
if embeddings.is_empty() {
|
||||
return Err(format!("index '{index_name}' has no embeddings — cannot bootstrap"));
|
||||
}
|
||||
|
||||
let mut eval = harness::synthetic_from_chunks(&name, index_name, &embeddings, 20, 10);
|
||||
harness::compute_ground_truth(&mut eval, &embeddings, &deps.ai_client)
|
||||
.await
|
||||
.map_err(|e| format!("auto-harness ground truth: {e}"))?;
|
||||
deps.harness_store.save(&eval).await
|
||||
.map_err(|e| format!("save auto-harness: {e}"))?;
|
||||
|
||||
tracing::info!(
|
||||
"agent: bootstrapped harness '{}' for index '{}' (20 self-queries, k=10)",
|
||||
name, index_name
|
||||
);
|
||||
Ok(name)
|
||||
}
|
||||
|
||||
async fn over_rate_limit(inner: &Arc<AgentInner>, cap: u32) -> bool {
|
||||
|
||||
@@ -28,7 +28,7 @@ use aibridge::client::AiClient;
|
||||
use catalogd::registry::Registry as CatalogRegistry;
|
||||
|
||||
use crate::embedding_cache::EmbeddingCache;
|
||||
use crate::harness;
|
||||
use crate::harness::{self, HarnessStore};
|
||||
use crate::hnsw::HnswStore;
|
||||
use crate::index_registry::IndexRegistry;
|
||||
use crate::jobs::JobTracker;
|
||||
@@ -128,7 +128,7 @@ fn pick_winner(trials: &[Trial], min_recall: f32) -> Option<&Trial> {
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub async fn run_autotune(
|
||||
req: AutotuneRequest,
|
||||
store: &Arc<dyn ObjectStore>,
|
||||
_store: &Arc<dyn ObjectStore>,
|
||||
catalog: &CatalogRegistry,
|
||||
ai_client: &AiClient,
|
||||
embedding_cache: &EmbeddingCache,
|
||||
@@ -136,6 +136,7 @@ pub async fn run_autotune(
|
||||
index_registry: &IndexRegistry,
|
||||
trial_journal: &TrialJournal,
|
||||
promotion_registry: &PromotionRegistry,
|
||||
harness_store: &HarnessStore,
|
||||
_job_tracker: &JobTracker,
|
||||
) -> Result<AutotuneResult, String> {
|
||||
let t0 = std::time::Instant::now();
|
||||
@@ -146,8 +147,11 @@ pub async fn run_autotune(
|
||||
}
|
||||
let _ = catalog; // reserved for future audit emission
|
||||
|
||||
// Load the harness once, compute ground truth once.
|
||||
let mut harness_set = harness::EvalSet::load(store, &req.harness)
|
||||
// Load the harness once, compute ground truth once. Harness resolves
|
||||
// to the index's bucket via HarnessStore, with primary as fallback for
|
||||
// pre-federation evals.
|
||||
let mut harness_set = harness_store
|
||||
.load_for_index(&req.index_name, &req.harness)
|
||||
.await
|
||||
.map_err(|e| format!("load harness: {e}"))?;
|
||||
|
||||
@@ -160,7 +164,7 @@ pub async fn run_autotune(
|
||||
harness::compute_ground_truth(&mut harness_set, &embeddings, ai_client)
|
||||
.await
|
||||
.map_err(|e| format!("ground truth: {e}"))?;
|
||||
harness_set.save(store).await.ok();
|
||||
harness_store.save(&harness_set).await.ok();
|
||||
}
|
||||
|
||||
let (grid, rejected) = sanitize_grid(req.grid.clone().unwrap_or_else(default_grid));
|
||||
|
||||
@@ -10,11 +10,14 @@
|
||||
use chrono::{DateTime, Utc};
|
||||
use object_store::ObjectStore;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::HashSet;
|
||||
use std::sync::Arc;
|
||||
|
||||
use aibridge::client::{AiClient, EmbedRequest};
|
||||
use storaged::ops;
|
||||
use storaged::registry::BucketRegistry;
|
||||
|
||||
use crate::index_registry::IndexRegistry;
|
||||
use crate::store::StoredEmbedding;
|
||||
|
||||
/// A single eval query with optional pre-computed ground truth.
|
||||
@@ -85,6 +88,108 @@ impl EvalSet {
|
||||
}
|
||||
}
|
||||
|
||||
/// Federation-aware wrapper around EvalSet persistence. Mirrors the
|
||||
/// `TrialJournal` / `PromotionRegistry` pattern: harness files colocate
|
||||
/// with their index's bucket (looked up via `IndexMeta.bucket`), falling
|
||||
/// back to `primary` for indexes the registry has never seen. Legacy
|
||||
/// harnesses predating federation remain discoverable — lookups transparently
|
||||
/// try the resolved bucket first, then `primary` as a fallback. Cross-bucket
|
||||
/// listing dedupes across every registered bucket so `GET /hnsw/evals`
|
||||
/// returns a complete picture.
|
||||
#[derive(Clone)]
|
||||
pub struct HarnessStore {
|
||||
buckets: Arc<BucketRegistry>,
|
||||
index_registry: IndexRegistry,
|
||||
}
|
||||
|
||||
impl HarnessStore {
|
||||
pub fn new(buckets: Arc<BucketRegistry>, index_registry: IndexRegistry) -> Self {
|
||||
Self { buckets, index_registry }
|
||||
}
|
||||
|
||||
/// Resolve which bucket holds this index's eval artifacts. Indexes the
|
||||
/// registry has never heard of fall through to `primary`.
|
||||
async fn bucket_for_index(&self, index_name: &str) -> String {
|
||||
self.index_registry
|
||||
.get(index_name)
|
||||
.await
|
||||
.map(|m| m.bucket)
|
||||
.unwrap_or_else(|| "primary".to_string())
|
||||
}
|
||||
|
||||
/// Save to the bucket that owns `eval.index_name`. Writes under the
|
||||
/// standard `_hnsw_evals/{name}.json` prefix of the resolved bucket.
|
||||
pub async fn save(&self, eval: &EvalSet) -> Result<(), String> {
|
||||
let bucket = self.bucket_for_index(&eval.index_name).await;
|
||||
let store = self.buckets.get(&bucket)?;
|
||||
eval.save(&store).await
|
||||
}
|
||||
|
||||
/// Load a harness by name, given the index it belongs to. Tries the
|
||||
/// index's bucket first; if the file is absent AND the resolved bucket
|
||||
/// isn't `primary`, falls through to `primary` so pre-federation evals
|
||||
/// remain reachable without migration.
|
||||
pub async fn load_for_index(
|
||||
&self,
|
||||
index_name: &str,
|
||||
harness_name: &str,
|
||||
) -> Result<EvalSet, String> {
|
||||
let bucket = self.bucket_for_index(index_name).await;
|
||||
let primary_store = self.buckets.get("primary")?;
|
||||
let store = self.buckets.get(&bucket)?;
|
||||
|
||||
match EvalSet::load(&store, harness_name).await {
|
||||
Ok(e) => Ok(e),
|
||||
Err(e) if bucket != "primary" => EvalSet::load(&primary_store, harness_name)
|
||||
.await
|
||||
.map_err(|primary_err| format!("{bucket}: {e}; primary fallback: {primary_err}")),
|
||||
Err(e) => Err(e),
|
||||
}
|
||||
}
|
||||
|
||||
/// Find a harness by name without knowing which index it belongs to —
|
||||
/// used by `GET /hnsw/evals/{name}`. Scans every registered bucket;
|
||||
/// first hit wins. Primary is searched first so pre-federation evals
|
||||
/// with the same name as a federated one resolve deterministically.
|
||||
pub async fn get_any(&self, harness_name: &str) -> Result<EvalSet, String> {
|
||||
let bucket_infos = self.buckets.list().await;
|
||||
let mut ordered: Vec<String> = bucket_infos.iter().map(|b| b.name.clone()).collect();
|
||||
ordered.sort_by_key(|n| if n == "primary" { 0 } else { 1 });
|
||||
|
||||
let mut last_err = None;
|
||||
for b in ordered {
|
||||
let store = match self.buckets.get(&b) {
|
||||
Ok(s) => s,
|
||||
Err(e) => { last_err = Some(e); continue; }
|
||||
};
|
||||
match EvalSet::load(&store, harness_name).await {
|
||||
Ok(e) => return Ok(e),
|
||||
Err(e) => { last_err = Some(e); }
|
||||
}
|
||||
}
|
||||
Err(last_err.unwrap_or_else(|| format!("no buckets registered for eval '{harness_name}'")))
|
||||
}
|
||||
|
||||
/// Union of every harness name across every registered bucket.
|
||||
/// Duplicates (same name in multiple buckets — pathological but
|
||||
/// possible after a manual migration) are collapsed.
|
||||
pub async fn list_all(&self) -> Vec<String> {
|
||||
let mut all: HashSet<String> = HashSet::new();
|
||||
for b in self.buckets.list().await {
|
||||
let store = match self.buckets.get(&b.name) {
|
||||
Ok(s) => s,
|
||||
Err(_) => continue,
|
||||
};
|
||||
if let Ok(names) = EvalSet::list(&store).await {
|
||||
all.extend(names);
|
||||
}
|
||||
}
|
||||
let mut out: Vec<String> = all.into_iter().collect();
|
||||
out.sort();
|
||||
out
|
||||
}
|
||||
}
|
||||
|
||||
/// Cosine similarity for two same-length f32 slices.
|
||||
fn cosine(a: &[f32], b: &[f32]) -> f32 {
|
||||
let mut dot = 0.0f32;
|
||||
@@ -127,13 +232,15 @@ pub fn recall_at_k(predicted: &[String], ground_truth: &[String], k: usize) -> f
|
||||
if ground_truth.is_empty() || k == 0 {
|
||||
return 0.0;
|
||||
}
|
||||
// Set-intersection recall@k. Previous implementation counted duplicates
|
||||
// in `predicted` (a corpus with repeated chunks — e.g. cached LLM
|
||||
// responses — returns the same doc_id multiple times via HNSW), which
|
||||
// inflated recall above 1.0 and poisoned promotion decisions.
|
||||
let gt_set: std::collections::HashSet<&String> =
|
||||
ground_truth.iter().take(k).collect();
|
||||
let hits = predicted
|
||||
.iter()
|
||||
.take(k)
|
||||
.filter(|d| gt_set.contains(d))
|
||||
.count();
|
||||
let pred_set: std::collections::HashSet<&String> =
|
||||
predicted.iter().take(k).collect();
|
||||
let hits = pred_set.intersection(>_set).count();
|
||||
hits as f32 / gt_set.len() as f32
|
||||
}
|
||||
|
||||
@@ -214,3 +321,67 @@ pub fn synthetic_from_chunks(
|
||||
ground_truth_built: false,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn s(v: &[&str]) -> Vec<String> { v.iter().map(|x| x.to_string()).collect() }
|
||||
|
||||
#[test]
|
||||
fn recall_empty_ground_truth_is_zero() {
|
||||
assert_eq!(recall_at_k(&s(&["a", "b"]), &[], 10), 0.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recall_k_zero_is_zero() {
|
||||
assert_eq!(recall_at_k(&s(&["a"]), &s(&["a"]), 0), 0.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recall_perfect_match_equals_one() {
|
||||
let pred = s(&["a", "b", "c"]);
|
||||
let gt = s(&["a", "b", "c"]);
|
||||
assert!((recall_at_k(&pred, >, 3) - 1.0).abs() < 1e-6);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recall_half_match() {
|
||||
let pred = s(&["a", "b", "x", "y"]);
|
||||
let gt = s(&["a", "b", "c", "d"]);
|
||||
assert!((recall_at_k(&pred, >, 4) - 0.5).abs() < 1e-6);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recall_duplicates_in_predicted_do_not_inflate() {
|
||||
// Regression guard: the previous implementation counted each
|
||||
// duplicate in `predicted` separately, which could push recall
|
||||
// above 1.0 on corpora with repeated chunks (cached responses etc).
|
||||
// Set-intersection semantics keep it bounded in [0, 1].
|
||||
let pred = s(&["a", "a", "a", "a"]);
|
||||
let gt = s(&["a", "b", "c", "d"]);
|
||||
let r = recall_at_k(&pred, >, 4);
|
||||
assert!(r <= 1.0, "recall {r} must not exceed 1.0");
|
||||
// One unique match out of four in gt = 0.25.
|
||||
assert!((r - 0.25).abs() < 1e-6);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recall_duplicates_in_ground_truth_handled() {
|
||||
// gt with dupes reduces effective |gt|; matching one hits all.
|
||||
let pred = s(&["x"]);
|
||||
let gt = s(&["x", "x", "x"]);
|
||||
let r = recall_at_k(&pred, >, 3);
|
||||
assert!(r <= 1.0);
|
||||
assert!((r - 1.0).abs() < 1e-6); // |pred ∩ gt| / |gt_set| = 1/1
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recall_respects_k_bound() {
|
||||
// k=2 means only the first 2 of pred and gt count toward the set.
|
||||
let pred = s(&["a", "b", "c", "d"]);
|
||||
let gt = s(&["a", "b", "c", "d"]);
|
||||
let r = recall_at_k(&pred, >, 2);
|
||||
assert!((r - 1.0).abs() < 1e-6);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -40,6 +40,12 @@ pub struct IndexMeta {
|
||||
/// `Lance` means it points at a Lance dataset directory.
|
||||
#[serde(default)]
|
||||
pub vector_backend: shared::types::VectorBackend,
|
||||
/// ADR-020: prefix prepended to doc_ids during embedding. If set,
|
||||
/// hybrid search strips this prefix to match against SQL primary keys.
|
||||
/// None = doc_ids ARE the raw primary keys (no stripping needed).
|
||||
/// Existing indexes: "W-", "CAND-", "W500K-", etc.
|
||||
#[serde(default)]
|
||||
pub id_prefix: Option<String>,
|
||||
}
|
||||
|
||||
fn default_bucket() -> String { "primary".to_string() }
|
||||
|
||||
@@ -22,7 +22,12 @@ pub struct Job {
|
||||
pub status: JobStatus,
|
||||
pub index_name: String,
|
||||
pub total_chunks: usize,
|
||||
/// How many chunks have been embedded so far.
|
||||
/// Also serialized as "processed" for backward-compat with monitoring scripts.
|
||||
pub embedded_chunks: usize,
|
||||
#[serde(rename = "processed")]
|
||||
pub processed_alias: usize,
|
||||
pub total: usize,
|
||||
pub progress_pct: f32,
|
||||
pub storage_key: Option<String>,
|
||||
pub error: Option<String>,
|
||||
@@ -53,6 +58,8 @@ impl JobTracker {
|
||||
index_name: index_name.to_string(),
|
||||
total_chunks,
|
||||
embedded_chunks: 0,
|
||||
processed_alias: 0,
|
||||
total: total_chunks,
|
||||
progress_pct: 0.0,
|
||||
storage_key: None,
|
||||
error: None,
|
||||
@@ -69,6 +76,7 @@ impl JobTracker {
|
||||
let mut jobs = self.jobs.write().await;
|
||||
if let Some(job) = jobs.get_mut(id) {
|
||||
job.embedded_chunks = embedded;
|
||||
job.processed_alias = embedded; // keep alias in sync
|
||||
job.progress_pct = if job.total_chunks > 0 {
|
||||
(embedded as f32 / job.total_chunks as f32) * 100.0
|
||||
} else {
|
||||
@@ -84,6 +92,7 @@ impl JobTracker {
|
||||
if let Some(job) = jobs.get_mut(id) {
|
||||
job.status = JobStatus::Completed;
|
||||
job.embedded_chunks = job.total_chunks;
|
||||
job.processed_alias = job.total_chunks;
|
||||
job.progress_pct = 100.0;
|
||||
job.storage_key = Some(storage_key);
|
||||
job.completed_at = Some(chrono::Utc::now().to_rfc3339());
|
||||
|
||||
@@ -7,6 +7,7 @@ pub mod harness;
|
||||
pub mod hnsw;
|
||||
pub mod index_registry;
|
||||
pub mod jobs;
|
||||
pub mod playbook_memory;
|
||||
pub mod promotion;
|
||||
pub mod refresh;
|
||||
pub mod store;
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
+114
-15
@@ -1,12 +1,92 @@
|
||||
/// RAG pipeline: question → embed → search → retrieve → generate answer.
|
||||
/// RAG pipeline: question → embed → search → rerank → generate answer.
|
||||
///
|
||||
/// The rerank step (added 2026-04-17) uses the LLM as a cross-encoder
|
||||
/// between retrieval and generation. This catches cases where the
|
||||
/// embedding model scores documents as similar but the content isn't
|
||||
/// actually relevant to the question — a known weakness of small
|
||||
/// general-purpose embed models on domain-specific text.
|
||||
|
||||
use object_store::ObjectStore;
|
||||
use std::sync::Arc;
|
||||
|
||||
use aibridge::client::{AiClient, EmbedRequest, GenerateRequest};
|
||||
use aibridge::continuation::{generate_continuable, ContinuableOpts, ResponseShape};
|
||||
use crate::search::{self, SearchResult};
|
||||
use crate::store;
|
||||
|
||||
/// Cross-encoder rerank: ask the LLM to re-sort retrieved chunks by
|
||||
/// relevance. Falls back to the original order if the model returns
|
||||
/// garbage (which happens ~5% of the time with 7B models).
|
||||
async fn rerank(
|
||||
question: &str,
|
||||
mut results: Vec<SearchResult>,
|
||||
ai_client: &AiClient,
|
||||
) -> Vec<SearchResult> {
|
||||
if results.len() <= 1 {
|
||||
return results;
|
||||
}
|
||||
|
||||
let chunk_list: String = results.iter().enumerate().map(|(i, r)| {
|
||||
let text: String = r.chunk_text.chars().take(200).collect();
|
||||
format!("[{i}] {text}")
|
||||
}).collect::<Vec<_>>().join("\n");
|
||||
|
||||
let resp = ai_client.generate(GenerateRequest {
|
||||
prompt: format!(
|
||||
"Rank these text chunks by relevance to the question.\n\
|
||||
Return ONLY a comma-separated list of indices, most relevant first.\n\n\
|
||||
Question: {question}\n\n\
|
||||
Chunks:\n{chunk_list}\n\n\
|
||||
Ranking:"
|
||||
),
|
||||
model: None,
|
||||
system: None,
|
||||
temperature: Some(0.0),
|
||||
max_tokens: Some(50),
|
||||
// Reranker returns a comma-separated int list — pure structured
|
||||
// output, zero benefit from hidden reasoning. Opt out to avoid
|
||||
// the empty-response failure mode Phase 21 catalogued.
|
||||
think: Some(false),
|
||||
}).await;
|
||||
|
||||
match resp {
|
||||
Ok(gen_resp) => {
|
||||
let text = gen_resp.text.trim();
|
||||
let indices: Vec<usize> = text
|
||||
.split(|c: char| c == ',' || c.is_whitespace())
|
||||
.filter_map(|s| s.trim().parse::<usize>().ok())
|
||||
.filter(|&i| i < results.len())
|
||||
.collect();
|
||||
|
||||
if indices.is_empty() {
|
||||
tracing::debug!("reranker returned unparseable output: {text}");
|
||||
return results;
|
||||
}
|
||||
|
||||
let mut reranked: Vec<SearchResult> = Vec::with_capacity(results.len());
|
||||
let mut used = vec![false; results.len()];
|
||||
for &i in &indices {
|
||||
if !used[i] {
|
||||
reranked.push(results[i].clone());
|
||||
used[i] = true;
|
||||
}
|
||||
}
|
||||
// Append any the model didn't mention (preserves all results).
|
||||
for (i, r) in results.drain(..).enumerate() {
|
||||
if !used[i] {
|
||||
reranked.push(r);
|
||||
}
|
||||
}
|
||||
tracing::info!("reranker reordered {}/{} chunks", indices.len(), reranked.len());
|
||||
reranked
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::debug!("reranker failed (using original order): {e}");
|
||||
results
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Full RAG answer with provenance.
|
||||
#[derive(Debug, Clone, serde::Serialize)]
|
||||
pub struct RagResponse {
|
||||
@@ -51,34 +131,53 @@ pub async fn query(
|
||||
});
|
||||
}
|
||||
|
||||
// 3. Build context from retrieved chunks
|
||||
// 3. Rerank: ask the LLM to sort retrieved chunks by relevance.
|
||||
// This cross-encoder step catches cases where embedding similarity
|
||||
// is high but semantic relevance is low — common with small embed
|
||||
// models on domain-specific text.
|
||||
let results = rerank(question, results, ai_client).await;
|
||||
|
||||
// 4. Build context from (reranked) top chunks
|
||||
let context: String = results.iter().enumerate().map(|(i, r)| {
|
||||
format!("[{}] (source: {}, doc: {}) {}", i + 1, r.source, r.doc_id, r.chunk_text)
|
||||
}).collect::<Vec<_>>().join("\n\n");
|
||||
|
||||
// 4. Generate answer
|
||||
// 5. Generate answer
|
||||
tracing::info!("RAG: generating answer from {} chunks", results.len());
|
||||
let prompt = format!(
|
||||
"You are a helpful assistant answering questions based on retrieved documents from a data system.\n\n\
|
||||
Use ONLY the following context to answer. If the context doesn't contain enough information, say so.\n\
|
||||
"You are a helpful assistant answering questions about a staffing database.\n\n\
|
||||
Use ONLY the following retrieved records to answer. Be specific — cite names,\n\
|
||||
numbers, skills, and cities from the records. If the context doesn't contain\n\
|
||||
enough information to fully answer, say what you can and note what's missing.\n\
|
||||
Cite sources by their number [1], [2], etc.\n\n\
|
||||
Context:\n{context}\n\n\
|
||||
Question: {question}\n\n\
|
||||
Answer:"
|
||||
);
|
||||
|
||||
let gen_resp = ai_client.generate(GenerateRequest {
|
||||
prompt,
|
||||
model: None,
|
||||
system: None,
|
||||
temperature: Some(0.2),
|
||||
max_tokens: Some(512),
|
||||
}).await?;
|
||||
// Route the answer call through Phase 21's generate_continuable so
|
||||
// a thinking-model empty-response or a mid-JSON truncation self-
|
||||
// repairs instead of silently returning half an answer. Shape is
|
||||
// Text (the answer is prose, not JSON), think is Some(false) to
|
||||
// opt out of hidden reasoning on the hot path. This is the first
|
||||
// production caller of the Phase 21 primitives — see audit finding
|
||||
// "Phase 21 Rust primitives are wired but not CALLED by any
|
||||
// production surface" from 2026-04-21.
|
||||
let mut cont_opts = ContinuableOpts::new("qwen2.5:latest");
|
||||
cont_opts.max_tokens = Some(512);
|
||||
cont_opts.temperature = Some(0.2);
|
||||
cont_opts.shape = ResponseShape::Text;
|
||||
cont_opts.think = Some(false);
|
||||
let outcome = generate_continuable(ai_client, &prompt, &cont_opts).await?;
|
||||
|
||||
Ok(RagResponse {
|
||||
answer: gen_resp.text.trim().to_string(),
|
||||
model: gen_resp.model,
|
||||
answer: outcome.text.trim().to_string(),
|
||||
// generate_continuable doesn't surface the model name (sidecar
|
||||
// echoes whatever Ollama loaded). Use the configured tier model
|
||||
// for now; if RAG needs to report the actual resolved model,
|
||||
// the runner can add a post-call ps probe later.
|
||||
model: "qwen2.5:latest".to_string(),
|
||||
sources: results,
|
||||
tokens_generated: gen_resp.tokens_generated,
|
||||
tokens_generated: None,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -307,6 +307,7 @@ async fn try_update_index_meta(
|
||||
chunks_per_sec: 0.0,
|
||||
bucket: "primary".to_string(),
|
||||
vector_backend: shared::types::VectorBackend::Parquet,
|
||||
id_prefix: None,
|
||||
};
|
||||
index_registry.register(meta).await
|
||||
}
|
||||
|
||||
+1195
-19
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,159 @@
|
||||
{
|
||||
"id": "07c119c3-2e59-463c-89a8-7980a849b6d6",
|
||||
"name": "workers_500k",
|
||||
"schema_fingerprint": "71686e497f5ee2fc77d50b13c39a46d89ca3d7673e9a0879e67c2db476764b25",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/workers_500k.parquet",
|
||||
"size_bytes": 323794818,
|
||||
"created_at": "2026-04-17T05:57:41.972944716Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T05:57:41.972945960Z",
|
||||
"updated_at": "2026-04-17T05:57:41.973093967Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": "pii",
|
||||
"columns": [
|
||||
{
|
||||
"name": "worker_id",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "role",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "email",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "phone",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "state",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "zip",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "skills",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "certifications",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "archetype",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "reliability",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "responsiveness",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "engagement",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "compliance",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "availability",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "communications",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "resume_text",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "csv",
|
||||
"source_file": "workers_500k.csv",
|
||||
"ingest_job": "ingest-1776405461972",
|
||||
"ingest_timestamp": "2026-04-17T05:57:41.972944716Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 500000,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,159 @@
|
||||
{
|
||||
"id": "0efdef2d-52bc-4154-82d4-056bb3a1915d",
|
||||
"name": "ethereal_workers",
|
||||
"schema_fingerprint": "9a1286ffada5390b459b217f56263e5ebffec1520ea29bab0be2efc6d5381adc",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/ethereal_workers.parquet",
|
||||
"size_bytes": 6991716,
|
||||
"created_at": "2026-04-17T03:23:29.399966984Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T03:23:29.399967772Z",
|
||||
"updated_at": "2026-04-17T03:23:29.400227081Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": "pii",
|
||||
"columns": [
|
||||
{
|
||||
"name": "worker_id",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "role",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "email",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "phone",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "state",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "zip",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "skills",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "certifications",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "archetype",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "reliability",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "responsiveness",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "engagement",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "compliance",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "availability",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "communications",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "resume_text",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "csv",
|
||||
"source_file": "ethereal_workers.csv",
|
||||
"ingest_job": "ingest-1776396209399",
|
||||
"ingest_timestamp": "2026-04-17T03:23:29.399966984Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 10000,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -1,23 +1,97 @@
|
||||
{
|
||||
"id": "0fd78303-9ad4-45fd-90d7-db95607d9ab1",
|
||||
"name": "timesheets",
|
||||
"schema_fingerprint": "auto",
|
||||
"schema_fingerprint": "806946533e133a4dd032f500644c9f702a46151fbb5bbd3b503c207dac86893b",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"bucket": "primary",
|
||||
"key": "datasets/timesheets.parquet",
|
||||
"size_bytes": 17539932,
|
||||
"created_at": "2026-03-27T14:42:43.922019299Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-27T14:42:43.922025703Z",
|
||||
"updated_at": "2026-03-27T14:42:43.922025703Z",
|
||||
"updated_at": "2026-04-17T02:45:58.282412872Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [],
|
||||
"columns": [
|
||||
{
|
||||
"name": "timesheet_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "placement_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "candidate_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "client_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "hours_regular",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "hours_overtime",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "bill_total",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "pay_total",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "week_ending",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "approved",
|
||||
"data_type": "Boolean",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": null
|
||||
"row_count": 1000000,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -1,23 +1,104 @@
|
||||
{
|
||||
"id": "1339f3d6-7677-47fb-8182-5f8e43f27cde",
|
||||
"name": "job_orders",
|
||||
"schema_fingerprint": "auto",
|
||||
"schema_fingerprint": "579a0593ca7c0e22ba53702821757f189077e1ddff7fd592c0097feed446a99a",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"bucket": "primary",
|
||||
"key": "datasets/job_orders.parquet",
|
||||
"size_bytes": 905534,
|
||||
"created_at": "2026-03-27T14:42:38.935718195Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-27T14:42:38.935724058Z",
|
||||
"updated_at": "2026-03-27T14:42:38.935724058Z",
|
||||
"updated_at": "2026-04-17T02:45:57.664902260Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [],
|
||||
"columns": [
|
||||
{
|
||||
"name": "job_order_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "client_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "title",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "vertical",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "bill_rate",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "pay_rate",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "status",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "state",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "zip",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "description",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": null
|
||||
"row_count": 15000,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"id": "15d889ef-2dba-4d1f-8ce0-22f7def35155",
|
||||
"name": "scanned_resume",
|
||||
"schema_fingerprint": "d3bc44bcafc990711442db3eecf25c0758232e3fccf6345200b1813fae5868c2",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/scanned_resume.parquet",
|
||||
"size_bytes": 3097,
|
||||
"created_at": "2026-04-17T01:44:33.927277187Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T01:44:33.927280048Z",
|
||||
"updated_at": "2026-04-17T01:44:33.927528900Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "source_file",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "page_number",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "text_content",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "pdf",
|
||||
"source_file": "test_scanned_resume.pdf",
|
||||
"ingest_job": "ingest-1776390273927",
|
||||
"ingest_timestamp": "2026-04-17T01:44:33.927277187Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 1,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"id": "17119168-cfd2-43b2-a4dc-ff2a2c5fd086",
|
||||
"name": "demo_customers",
|
||||
"schema_fingerprint": "7af9708f2366f790e0ae03db6f681f5a4d4b8af4d1527b091bedfee1be81d304",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/demo_customers.parquet",
|
||||
"size_bytes": 4583,
|
||||
"created_at": "2026-04-17T01:35:00.165218923Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T01:35:00.165220510Z",
|
||||
"updated_at": "2026-04-17T05:56:44.964702014Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "email",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "tier",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "lifetime_spend",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "is_active",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "signed_up_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "notes",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 11,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -1,23 +1,132 @@
|
||||
{
|
||||
"id": "1d8a065e-59c1-45ce-967b-398bc8370cbb",
|
||||
"name": "candidates",
|
||||
"schema_fingerprint": "auto",
|
||||
"schema_fingerprint": "206360ab312e4c13679ed0ab4ccb3702250cc3e2176cc669d41471584c556c73",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"bucket": "primary",
|
||||
"key": "datasets/candidates.parquet",
|
||||
"size_bytes": 10592165,
|
||||
"created_at": "2026-03-27T14:42:38.823368759Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-27T14:42:38.823374843Z",
|
||||
"updated_at": "2026-03-27T14:42:38.823374843Z",
|
||||
"updated_at": "2026-04-17T02:45:57.722237378Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [],
|
||||
"columns": [
|
||||
{
|
||||
"name": "candidate_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "first_name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "last_name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "email",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "phone",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "state",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "zip",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "vertical",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "skills",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "resume_summary",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "status",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "source",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "min_pay_rate",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "years_experience",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": null
|
||||
"row_count": 100000,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
{
|
||||
"id": "2b8e8b01-a44e-49c5-a103-a1b55898bee1",
|
||||
"name": "kb_memory_entries",
|
||||
"schema_fingerprint": "15dbebd0abb906577e11cbb73083abeb8961b8c603a2472cafa8c46100c3fb1c",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/kb_memory_entries.parquet",
|
||||
"size_bytes": 8795,
|
||||
"created_at": "2026-04-17T05:22:06.356227657Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T05:22:06.356228933Z",
|
||||
"updated_at": "2026-04-17T05:22:06.356776285Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "content",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "content_type",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "source",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "session_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "embedding",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "metadata",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "importance_score",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "created_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "accessed_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "access_count",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "postgresql",
|
||||
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
|
||||
"ingest_job": "pg-stream-1776403326356",
|
||||
"ingest_timestamp": "2026-04-17T05:22:06.356227657Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 13,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
{
|
||||
"id": "2db2d92f-1171-4f2f-a707-ad387b6cd464",
|
||||
"name": "onboard_live_1776729919",
|
||||
"schema_fingerprint": "9cdf2fc924bad875e63bd12a82dfb1cd080c5cbba71f671b4a2d41e746df187d",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/onboard_live_1776729919.parquet",
|
||||
"size_bytes": 31444,
|
||||
"created_at": "2026-04-21T00:05:19.717239967Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-21T00:05:19.717241621Z",
|
||||
"updated_at": "2026-04-21T00:05:19.717370050Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": "pii",
|
||||
"columns": [
|
||||
{
|
||||
"name": "worker_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "role",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "state",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "email",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "phone",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "skills",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "certifications",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "availability",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "reliability",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "archetype",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "csv",
|
||||
"source_file": "sample_1776729919.csv",
|
||||
"ingest_job": "ingest-1776729919717",
|
||||
"ingest_timestamp": "2026-04-21T00:05:19.717239967Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 173,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
{
|
||||
"id": "3178f0da-5985-4731-b343-162dd415f3be",
|
||||
"name": "kb_self_reports",
|
||||
"schema_fingerprint": "76382b028b92bb0b361306dced3d773af0ee8de6aa160cedfc4d649f2860167a",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/kb_self_reports.parquet",
|
||||
"size_bytes": 51134,
|
||||
"created_at": "2026-04-17T04:14:34.355419647Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T04:14:34.355420701Z",
|
||||
"updated_at": "2026-04-17T06:06:21.358604060Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "report_type",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "model",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "report",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "data_size",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "created_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "postgresql",
|
||||
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
|
||||
"ingest_job": "pg-stream-1776399274355",
|
||||
"ingest_timestamp": "2026-04-17T04:14:34.355419647Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 11,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"id": "32ee74a0-59b4-4e5b-8edb-70c9347a4bf3",
|
||||
"name": "successful_playbooks_live",
|
||||
"schema_fingerprint": "4e28e94650d9f5689e952674aebfd09db70edfcba617713081c0a83bf59121ce",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/successful_playbooks_live.parquet",
|
||||
"size_bytes": 284341,
|
||||
"created_at": "2026-04-22T03:28:28.343840136Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-20T11:07:57.308050648Z",
|
||||
"updated_at": "2026-04-22T03:28:28.343843823Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": null,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,159 @@
|
||||
{
|
||||
"id": "33544879-fcfa-4458-b782-a4ca8dafed08",
|
||||
"name": "workers_100k",
|
||||
"schema_fingerprint": "1cc2a0caf174aa9bf67b1e2cb7858a312a0c94d604d906f807eb5610864ff70e",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/workers_100k.parquet",
|
||||
"size_bytes": 65225188,
|
||||
"created_at": "2026-04-17T04:54:03.306737596Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T04:54:03.306740279Z",
|
||||
"updated_at": "2026-04-17T04:54:03.306927852Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": "pii",
|
||||
"columns": [
|
||||
{
|
||||
"name": "worker_id",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "role",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "email",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "phone",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "state",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "zip",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "skills",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "certifications",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "archetype",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "reliability",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "responsiveness",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "engagement",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "compliance",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "availability",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "communications",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "resume_text",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "csv",
|
||||
"source_file": "workers_100k.csv",
|
||||
"ingest_job": "ingest-1776401643306",
|
||||
"ingest_timestamp": "2026-04-17T04:54:03.306737596Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 100000,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"id": "35f36c38-5cd3-438e-b93d-7ff7720f0346",
|
||||
"name": "kb_meta_runs",
|
||||
"schema_fingerprint": "68f2c0d7a3ceb0aaa3c17c64900704519c72d213161bc9e5179c42ee53f6d0df",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/kb_meta_runs.parquet",
|
||||
"size_bytes": 886387,
|
||||
"created_at": "2026-04-17T05:22:06.314413387Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T05:22:06.314414515Z",
|
||||
"updated_at": "2026-04-17T06:06:21.300610797Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "pipeline_id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "iteration",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "stage_results",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "final_output",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "score",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "model_config",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "duration_ms",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "created_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 48,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -4,7 +4,7 @@
|
||||
"schema_fingerprint": "30c0e31f0963e6f4af02131bbb9ea246fbbd068b849b833565a4b28211fbc90b",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"bucket": "primary",
|
||||
"key": "datasets/users.parquet",
|
||||
"size_bytes": 2012,
|
||||
"created_at": "2026-03-28T01:38:59.904968123Z"
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
"schema_fingerprint": "68f2c0d7a3ceb0aaa3c17c64900704519c72d213161bc9e5179c42ee53f6d0df",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"bucket": "primary",
|
||||
"key": "datasets/meta_runs.parquet",
|
||||
"size_bytes": 729773,
|
||||
"created_at": "2026-03-28T01:38:57.380576453Z"
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
{
|
||||
"id": "564b00ae-cbf3-4efd-aa55-84cdb6d2b0b7",
|
||||
"name": "client_workerskjkk",
|
||||
"schema_fingerprint": "cdfe85348885ddf329e5e6e9bf0e2c75c92d1a86fdb0fd3875ed46e3f93c4d82",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/client_workerskjkk.parquet",
|
||||
"size_bytes": 32201,
|
||||
"created_at": "2026-04-21T00:49:04.623625149Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-21T00:49:04.623626738Z",
|
||||
"updated_at": "2026-04-21T00:49:04.623901788Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": "pii",
|
||||
"columns": [
|
||||
{
|
||||
"name": "worker_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "role",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "state",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "email",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "phone",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "skills",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "certifications",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "availability",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "reliability",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "archetype",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "csv",
|
||||
"source_file": "staffing_roster_sample.csv",
|
||||
"ingest_job": "ingest-1776732544623",
|
||||
"ingest_timestamp": "2026-04-21T00:49:04.623625149Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 180,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"id": "57e557a8-4754-47c9-9636-e01d39fc20f2",
|
||||
"name": "animals",
|
||||
"schema_fingerprint": "b024b907f151889aca5407fc46fe1821d2600f587cb1a26742843597a66436f1",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/animals.parquet",
|
||||
"size_bytes": 1107,
|
||||
"created_at": "2026-04-16T13:47:57.033455363Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-16T13:47:57.033456007Z",
|
||||
"updated_at": "2026-04-16T13:47:57.033631067Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "animal_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "species",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "habitat",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "csv",
|
||||
"source_file": "animals.csv",
|
||||
"ingest_job": "ingest-1776347277033",
|
||||
"ingest_timestamp": "2026-04-16T13:47:57.033455363Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 3,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
{
|
||||
"id": "5a35646f-b357-4636-b88f-203aba0d7435",
|
||||
"name": "people_test",
|
||||
"schema_fingerprint": "bba35b6cd1e399dc0504c760c767ab391f7ed81a41573c15b3bd4eb0e3bcc382",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "testing",
|
||||
"key": "datasets/people_test.parquet",
|
||||
"size_bytes": 1402,
|
||||
"created_at": "2026-04-16T13:47:32.669517398Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-16T13:47:32.669518210Z",
|
||||
"updated_at": "2026-04-16T13:47:32.669787451Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": "pii",
|
||||
"columns": [
|
||||
{
|
||||
"name": "person_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "role",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "csv",
|
||||
"source_file": "people.csv",
|
||||
"ingest_job": "ingest-1776347252669",
|
||||
"ingest_timestamp": "2026-04-16T13:47:32.669517398Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 3,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
{
|
||||
"id": "5bab2fac-8bdd-41b7-a993-ed48fecc60b3",
|
||||
"name": "onboard_demo2",
|
||||
"schema_fingerprint": "274575423f7d2648cea49a628a2e827dbd8822cef1f4eebfcf8e0a5f8602fbf7",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/onboard_demo2.parquet",
|
||||
"size_bytes": 7833,
|
||||
"created_at": "2026-04-20T23:13:36.886222056Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-20T23:13:36.886223472Z",
|
||||
"updated_at": "2026-04-20T23:13:36.886426834Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": "pii",
|
||||
"columns": [
|
||||
{
|
||||
"name": "worker_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "role",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "state",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "email",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "phone",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "skills",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "certifications",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "availability",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "reliability",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "archetype",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "csv",
|
||||
"source_file": "staffing_roster_sample.csv",
|
||||
"ingest_job": "ingest-1776726816886",
|
||||
"ingest_timestamp": "2026-04-20T23:13:36.886222056Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 25,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
{
|
||||
"id": "909a1fd1-0924-4a5e-a527-d249a07540a1",
|
||||
"name": "kb_response_cache",
|
||||
"schema_fingerprint": "c90d7be310b5025d2c4d398cf07692d8d9bb46ed591c0a87b339bafcac9ddeed",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/kb_response_cache.parquet",
|
||||
"size_bytes": 8464841,
|
||||
"created_at": "2026-04-17T05:22:06.168071783Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T05:22:06.168072339Z",
|
||||
"updated_at": "2026-04-17T06:06:21.356504054Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "cache_key",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "prompt",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "mode",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "models_used",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "best_run_id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "best_score",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "responses",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "hit_count",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "created_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "updated_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 197,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -1,23 +1,69 @@
|
||||
{
|
||||
"id": "94a8bd16-6756-43af-b951-09a9e6b8300f",
|
||||
"name": "clients",
|
||||
"schema_fingerprint": "auto",
|
||||
"schema_fingerprint": "1ba20a49d967c85c41640a9b9a61287ae9fa483daf6c71a197591fe3228c76d2",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"bucket": "primary",
|
||||
"key": "datasets/clients.parquet",
|
||||
"size_bytes": 21971,
|
||||
"created_at": "2026-03-27T14:42:38.830329102Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-27T14:42:38.830331694Z",
|
||||
"updated_at": "2026-03-27T14:42:38.830331694Z",
|
||||
"updated_at": "2026-04-17T02:45:58.138164775Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [],
|
||||
"columns": [
|
||||
{
|
||||
"name": "client_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "company_name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "vertical",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "state",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "zip",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": null
|
||||
"row_count": 2000,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
{
|
||||
"id": "9861b817-f571-496b-9d7b-c4d385e51ca5",
|
||||
"name": "sparse_workers",
|
||||
"schema_fingerprint": "1055d5e17fa3285a1cac7d4c949ae03699cd046af671f22d0d1cf4d0024c66a4",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/sparse_workers.parquet",
|
||||
"size_bytes": 7619,
|
||||
"created_at": "2026-04-17T20:31:24.968688194Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T20:31:24.968689336Z",
|
||||
"updated_at": "2026-04-17T20:31:24.968969734Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": "pii",
|
||||
"columns": [
|
||||
{
|
||||
"name": "name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "phone",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "role",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "state",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "notes",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "csv",
|
||||
"source_file": "sparse_workers.csv",
|
||||
"ingest_job": "ingest-1776457884968",
|
||||
"ingest_timestamp": "2026-04-17T20:31:24.968688194Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 200,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -1,23 +1,83 @@
|
||||
{
|
||||
"id": "9c4d9116-1d9d-4afd-a8d1-c514a678e5fa",
|
||||
"name": "call_log",
|
||||
"schema_fingerprint": "auto",
|
||||
"schema_fingerprint": "d1f82ed776afc484747f8f58b30ccd00d10bca73703b1479a67647f98eeff7e2",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"bucket": "primary",
|
||||
"key": "datasets/call_log.parquet",
|
||||
"size_bytes": 35951077,
|
||||
"created_at": "2026-03-27T14:42:47.395548205Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-27T14:42:47.395555326Z",
|
||||
"updated_at": "2026-03-27T14:42:47.395555326Z",
|
||||
"updated_at": "2026-04-17T02:45:58.137293143Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [],
|
||||
"columns": [
|
||||
{
|
||||
"name": "call_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "from_number",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "to_number",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "duration_seconds",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "timestamp",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "recruiter",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "candidate_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "disposition",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": null
|
||||
"row_count": 800000,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,237 @@
|
||||
{
|
||||
"id": "a2206c79-28d6-40f2-b97b-61a5df5bee06",
|
||||
"name": "kb_threat_intel",
|
||||
"schema_fingerprint": "df1e126046147b3de42086880e10c3501a3a615ecddf336bc24957a24c321241",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/kb_threat_intel.parquet",
|
||||
"size_bytes": 247318,
|
||||
"created_at": "2026-04-17T04:23:16.149697926Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T04:23:16.149698245Z",
|
||||
"updated_at": "2026-04-17T05:56:44.994303201Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "ip",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "threat_level",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "classification",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "confidence",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "summary",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "indicators",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "recommendation",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "pattern",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "attack_type",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "likely_automated",
|
||||
"data_type": "Boolean",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "country",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "country_code",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "isp",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "org",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "asn",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "is_proxy",
|
||||
"data_type": "Boolean",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "is_hosting",
|
||||
"data_type": "Boolean",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "open_ports",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "blocklist_count",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "blocklist_total",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "blocklists_blocked",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "reverse_dns",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "traceroute",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "log_count",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "banned",
|
||||
"data_type": "Boolean",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "enriched_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "updated_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "raw_data",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 54,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
{
|
||||
"id": "a99da7e2-8f42-4ff3-8fa7-679d4e514581",
|
||||
"name": "kb_response_cache_history",
|
||||
"schema_fingerprint": "9a05c209a51f9543bd7dc9387695b0c67a6abc135b53eaeec458140712bfcf50",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/kb_response_cache_history.parquet",
|
||||
"size_bytes": 8396568,
|
||||
"created_at": "2026-04-17T04:14:34.260241351Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T04:14:34.260242474Z",
|
||||
"updated_at": "2026-04-17T06:06:21.334716946Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "cache_key",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "run_id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "score",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "responses",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "models_used",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "recorded_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "postgresql",
|
||||
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
|
||||
"ingest_job": "pg-stream-1776403326260",
|
||||
"ingest_timestamp": "2026-04-17T05:22:06.260249210Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 207,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
{
|
||||
"id": "c2f0acdb-df65-4c21-9633-08af4248c85b",
|
||||
"name": "kb_lab_trials",
|
||||
"schema_fingerprint": "1d5782349402439a7e44efd0ccab9ae64ac3044221adef9e828b60b8bbb44dd5",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/kb_lab_trials.parquet",
|
||||
"size_bytes": 68221,
|
||||
"created_at": "2026-04-17T05:22:06.382106104Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T05:22:06.382107354Z",
|
||||
"updated_at": "2026-04-17T06:06:21.358304760Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "experiment_id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "trial_num",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "config_diff",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "config_snapshot",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "scores",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "avg_score",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "improved",
|
||||
"data_type": "Boolean",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "duration_ms",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "created_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 41,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -1,23 +1,83 @@
|
||||
{
|
||||
"id": "d35c7941-37e2-4bde-8226-5cf69c74931a",
|
||||
"name": "placements",
|
||||
"schema_fingerprint": "auto",
|
||||
"schema_fingerprint": "962535a1810c07183a882c15a76a11a02ad3afef844e71975b0eaa48943487f6",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"bucket": "primary",
|
||||
"key": "datasets/placements.parquet",
|
||||
"size_bytes": 1213820,
|
||||
"created_at": "2026-03-27T14:42:39.040983450Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-27T14:42:39.040989351Z",
|
||||
"updated_at": "2026-03-27T14:42:39.040989351Z",
|
||||
"updated_at": "2026-04-17T02:45:58.147407036Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [],
|
||||
"columns": [
|
||||
{
|
||||
"name": "placement_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "candidate_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "job_order_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "client_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "bill_rate",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "pay_rate",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "recruiter",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "status",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": null
|
||||
"row_count": 50000,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
{
|
||||
"id": "d668a036-7b50-4845-8194-6a03ef370c8d",
|
||||
"name": "kb_pipeline_runs",
|
||||
"schema_fingerprint": "c019b81feb58ff2aefe4cbe700056a100e25716f2aa8e8415ba8f20656812f75",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/kb_pipeline_runs.parquet",
|
||||
"size_bytes": 1518415,
|
||||
"created_at": "2026-04-17T05:22:06.289208160Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T05:22:06.289209657Z",
|
||||
"updated_at": "2026-04-17T06:06:21.311384739Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "pipeline",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "topic",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "status",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "steps",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "result",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "models_used",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "duration_ms",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "created_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "completed_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 195,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
{
|
||||
"id": "d92c5d57-14f9-4ce0-9c4f-19c6db94a170",
|
||||
"name": "kb_team_runs",
|
||||
"schema_fingerprint": "d704ee2b9b434774aed2258da2fddcdcbab226547a011ba24d4281253657bdd3",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/kb_team_runs.parquet",
|
||||
"size_bytes": 19293718,
|
||||
"created_at": "2026-04-17T05:22:06.071898629Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T05:22:06.071899235Z",
|
||||
"updated_at": "2026-04-17T05:56:44.992569105Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "mode",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "prompt",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "config",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "responses",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "models_used",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "created_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "archived",
|
||||
"data_type": "Boolean",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "tags",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "notes",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "quality_score",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "score_method",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "score_metadata",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 590,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
{
|
||||
"id": "dab56959-d79b-4c68-a6bd-e5fc487621ff",
|
||||
"name": "successful_playbooks",
|
||||
"schema_fingerprint": "30b52caca7b6303852cd648882eae9ed87205b80e5e570f6fda08dc8735d9b06",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/successful_playbooks.parquet",
|
||||
"size_bytes": 2540,
|
||||
"created_at": "2026-04-17T22:25:34.467634304Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T22:25:34.467635066Z",
|
||||
"updated_at": "2026-04-19T08:44:08.533058251Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "timestamp",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "operation",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "approach",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "result",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "context",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "csv",
|
||||
"source_file": "playbook.csv",
|
||||
"ingest_job": "ingest-1776588248532",
|
||||
"ingest_timestamp": "2026-04-19T08:44:08.532686879Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 29,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -4,7 +4,7 @@
|
||||
"schema_fingerprint": "1d5782349402439a7e44efd0ccab9ae64ac3044221adef9e828b60b8bbb44dd5",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"bucket": "primary",
|
||||
"key": "datasets/lab_trials.parquet",
|
||||
"size_bytes": 64646,
|
||||
"created_at": "2026-03-28T01:14:03.026116573Z"
|
||||
|
||||
@@ -1,23 +1,83 @@
|
||||
{
|
||||
"id": "e959ca11-9f6b-4843-864a-cc3f50a8aa60",
|
||||
"name": "email_log",
|
||||
"schema_fingerprint": "auto",
|
||||
"schema_fingerprint": "b28cbe711bba32611306f9a3186f67c1510c3e39b0c8812d765fe2b44bd65634",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"bucket": "primary",
|
||||
"key": "datasets/email_log.parquet",
|
||||
"size_bytes": 16768671,
|
||||
"created_at": "2026-03-27T14:42:49.271082991Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-27T14:42:49.271091077Z",
|
||||
"updated_at": "2026-03-27T14:42:49.271091077Z",
|
||||
"updated_at": "2026-04-17T02:45:57.881479855Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [],
|
||||
"columns": [
|
||||
{
|
||||
"name": "email_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "from_addr",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "to_addr",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "subject",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "timestamp",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "recruiter",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "candidate_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "opened",
|
||||
"data_type": "Boolean",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": null
|
||||
"row_count": 500000,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
{
|
||||
"id": "eeb0d288-e227-478d-9407-78a43b623c5c",
|
||||
"name": "kb_meta_pipelines",
|
||||
"schema_fingerprint": "cabe1d7fc442e1dfcfaabf663509d590c6edc84b445a91acaf0ae68b94aff518",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/kb_meta_pipelines.parquet",
|
||||
"size_bytes": 32085,
|
||||
"created_at": "2026-04-17T04:14:34.315211960Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T04:14:34.315212764Z",
|
||||
"updated_at": "2026-04-17T06:06:21.357779083Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "data_source",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "stages",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "status",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "current_stage",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "total_stages",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "results",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "best_score",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "iterations",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "config",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "created_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "updated_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 6,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
+11
-14
@@ -1,17 +1,17 @@
|
||||
{
|
||||
"id": "e7304f05-5278-4e17-961a-51f2588fd2aa",
|
||||
"id": "ef6c0acf-de7a-4be0-877e-e0c2889f390f",
|
||||
"name": "threat_intel",
|
||||
"schema_fingerprint": "df1e126046147b3de42086880e10c3501a3a615ecddf336bc24957a24c321241",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"bucket": "primary",
|
||||
"key": "datasets/threat_intel.parquet",
|
||||
"size_bytes": 111130,
|
||||
"created_at": "2026-03-28T01:14:03.054140697Z"
|
||||
"size_bytes": 247112,
|
||||
"created_at": "2026-04-19T11:19:40.517865325Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-28T01:14:03.054141294Z",
|
||||
"updated_at": "2026-03-28T01:14:03.054427047Z",
|
||||
"created_at": "2026-04-16T07:59:13.882669807Z",
|
||||
"updated_at": "2026-04-19T11:19:40.517873046Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
@@ -227,14 +227,11 @@
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "postgresql",
|
||||
"source_file": "127.0.0.1:5432/knowledge_base.threat_intel",
|
||||
"ingest_job": "pg-import-1774660443054",
|
||||
"ingest_timestamp": "2026-03-28T01:14:03.054140697Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 20
|
||||
"row_count": 20,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -4,7 +4,7 @@
|
||||
"schema_fingerprint": "0fcee1ac176b9bc45ab392bbd4401042803eb80d0de48b1fa6a5e20ffb27fa8d",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"bucket": "primary",
|
||||
"key": "datasets/new_candidates.parquet",
|
||||
"size_bytes": 2731,
|
||||
"created_at": "2026-03-28T01:04:22.787759218Z"
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
{
|
||||
"id": "f2fda821-2f86-4c81-af58-488d184d8243",
|
||||
"name": "resumes",
|
||||
"schema_fingerprint": "a2b2c191412faf317800637da5887d30d749d5ba0222d0a8edd12cde08daeb8a",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/resumes.parquet",
|
||||
"size_bytes": 3159,
|
||||
"created_at": "2026-04-19T11:26:29.607950059Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-19T11:26:29.607951447Z",
|
||||
"updated_at": "2026-04-19T11:26:29.608054435Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "doc_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "resume_text",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "ndjson",
|
||||
"source_file": "resumes.ndjson",
|
||||
"ingest_job": "ingest-1776597989607",
|
||||
"ingest_timestamp": "2026-04-19T11:26:29.607950059Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 10,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,107 @@
|
||||
# ADR-020: Universal ID Mapping — the hybrid search identity problem
|
||||
|
||||
**Status:** Proposed — 2026-04-17
|
||||
**Triggered by:** ID mismatch between vector doc_ids and SQL primary keys
|
||||
**Owner:** J
|
||||
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
The hybrid search endpoint (`POST /vectors/hybrid`) filters SQL results
|
||||
by primary key, then matches those keys against vector doc_ids. This
|
||||
requires the two ID spaces to be compatible. Currently they're not:
|
||||
|
||||
| Source | SQL primary key | Vector doc_id | Match? |
|
||||
|---|---|---|---|
|
||||
| ethereal_workers | `worker_id = 4925` | `W-4925` | Only with W- strip |
|
||||
| workers_500k | `worker_id = 41566` | `W500K-41566` | Only with W500K- strip |
|
||||
| candidates | `candidate_id = CAND-055035` | `CAND-055035` | Yes (same format) |
|
||||
| workers_5k_proof | `worker_id = 3200` | `W5K-3200` | Only with W5K- strip |
|
||||
|
||||
Every time a new dataset is ingested and embedded, a new prefix
|
||||
appears and the hybrid search breaks until someone hardcodes another
|
||||
`strip_prefix` line.
|
||||
|
||||
This violates PRD invariant: **"Any data source can be ingested
|
||||
without pre-defined schemas."** The ID format IS a pre-defined schema
|
||||
that the hybrid search depends on.
|
||||
|
||||
## Root cause
|
||||
|
||||
When the embedding pipeline creates a vector index, it generates
|
||||
doc_ids by concatenating a prefix + the source row's ID column. The
|
||||
prefix is chosen at index creation time and baked into the Parquet
|
||||
vector file. The SQL dataset knows nothing about this prefix.
|
||||
|
||||
The hybrid search then has to reverse-engineer the prefix to match
|
||||
vector results against SQL rows. This is fragile and breaks on every
|
||||
new data source.
|
||||
|
||||
## Solution: catalog-level ID mapping
|
||||
|
||||
### Option A: Normalize doc_ids at embedding time (simplest)
|
||||
|
||||
When creating a vector index, don't prefix the doc_id. Use the raw
|
||||
value from the source column. If the source has `worker_id = 4925`,
|
||||
the vector doc_id is just `"4925"` — no `W-`, no `W500K-`.
|
||||
|
||||
**Pros:** Simplest. Hybrid search just compares strings directly.
|
||||
**Cons:** Doc_ids across different indexes could collide (two datasets
|
||||
both have worker_id=1). Need to scope by index name.
|
||||
|
||||
### Option B: Catalog stores the mapping (most robust)
|
||||
|
||||
The catalog maintains a mapping table:
|
||||
```
|
||||
index_name | doc_id_prefix | source_dataset | source_id_column
|
||||
-----------+---------------+----------------+-----------------
|
||||
resumes | CAND- | candidates | candidate_id
|
||||
workers_v1 | W500K- | workers_500k | worker_id
|
||||
```
|
||||
|
||||
Hybrid search reads this mapping and applies the prefix/strip
|
||||
automatically. New datasets register their mapping at index creation.
|
||||
|
||||
**Pros:** Handles all cases. Self-describing. No code changes for new data.
|
||||
**Cons:** One more lookup per hybrid search (trivial perf cost).
|
||||
|
||||
### Option C: Pass the mapping in the request (pragmatic)
|
||||
|
||||
The hybrid search request already accepts `id_column`. Extend it
|
||||
with `id_prefix` so the caller says "vector doc_ids have prefix
|
||||
W500K- and the SQL column is worker_id."
|
||||
|
||||
```json
|
||||
{
|
||||
"index_name": "workers_500k_v1",
|
||||
"id_column": "worker_id",
|
||||
"id_prefix": "W500K-",
|
||||
"sql_filter": "role = 'Forklift Operator' AND state = 'IL'",
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
**Pros:** Zero backend change. Caller already knows the context.
|
||||
**Cons:** Caller has to know the prefix — not self-service.
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Option A for new indexes, Option B for registry.**
|
||||
|
||||
1. Change the embedding pipeline to use RAW IDs (no prefix) as the
|
||||
default. Existing indexes keep their prefixed IDs.
|
||||
2. Add an `id_prefix` column to IndexMeta so the catalog knows how
|
||||
to map between vector and SQL IDs.
|
||||
3. Hybrid search reads IndexMeta.id_prefix and applies it automatically.
|
||||
4. Remove the hardcoded strip_prefix chain.
|
||||
|
||||
This means: ingest a new dataset, embed it, hybrid search works
|
||||
immediately. No code changes. The system is truly "any data source."
|
||||
|
||||
## Implementation
|
||||
|
||||
1. `IndexMeta` gains `id_prefix: Option<String>` (default None = no prefix)
|
||||
2. Embedding pipeline: when `id_prefix` is None, use raw IDs from source
|
||||
3. Hybrid search: read `IndexMeta.id_prefix`, prepend to SQL IDs before matching
|
||||
4. Migration: existing indexes retain their prefix in IndexMeta
|
||||
@@ -0,0 +1,93 @@
|
||||
# ADR-021: Sparse Data Trust Path — start with nothing, earn everything
|
||||
|
||||
**Status:** Proposed — 2026-04-17
|
||||
**Triggered by:** Legacy staffing company pushback: "we don't have that data"
|
||||
**Owner:** J
|
||||
|
||||
---
|
||||
|
||||
## The Problem
|
||||
|
||||
We demonstrated a system with rich worker profiles (18 fields,
|
||||
behavioral scores, certifications, communication history). The
|
||||
staffing company said: "We don't have any of that. We have a name,
|
||||
a phone number, and a contract."
|
||||
|
||||
They're right. Our demo assumed data that doesn't exist in their
|
||||
world. Showing AI that only works with perfect data is worse than
|
||||
useless — it builds distrust.
|
||||
|
||||
## Their Reality
|
||||
|
||||
Day 1 data for a typical worker:
|
||||
- Name
|
||||
- Phone number
|
||||
- Maybe: city, role, one or two skills mentioned on a call
|
||||
|
||||
Day 1 data for a contract:
|
||||
- Client name
|
||||
- Role needed
|
||||
- Headcount
|
||||
- Start date
|
||||
- Maybe: required certs, location
|
||||
|
||||
That's it. No reliability scores. No availability metrics. No
|
||||
communication history. No certifications in a database. The staffing
|
||||
coordinator carries that knowledge in their HEAD.
|
||||
|
||||
## The Trust Path
|
||||
|
||||
### Phase 1: Work with what they have (Day 1)
|
||||
- Accept sparse profiles: name + phone + role. That's enough.
|
||||
- Match contracts to workers by role + location only. No scores.
|
||||
- The system is useful immediately: "here are the 12 people you
|
||||
have tagged as Forklift Operators in IL." That's already faster
|
||||
than scrolling a spreadsheet.
|
||||
- Don't show empty fields. Don't show 0% bars. Don't show what's
|
||||
missing — show what's THERE.
|
||||
|
||||
### Phase 2: Earn data through use (Week 1-4)
|
||||
- Every placement generates a timesheet → reliability starts building
|
||||
- Every call logged → communication history accumulates
|
||||
- Every check-in → availability becomes real
|
||||
- Every cert verified → certification database grows
|
||||
- The staffer doesn't "enter data" — they just do their job, and
|
||||
the system learns from it.
|
||||
|
||||
### Phase 3: AI starts helping (Month 2+)
|
||||
- Enough data to actually score workers → show reliability
|
||||
- Enough history to predict availability → surface it
|
||||
- Enough placements to know client preferences → suggest matches
|
||||
- The enrichment happened BECAUSE they used the system, not as a
|
||||
prerequisite TO use it.
|
||||
|
||||
## What This Means for the UI
|
||||
|
||||
- Worker cards must gracefully degrade. Name only? Show name only.
|
||||
Name + role? Show name + role. Full profile? Show everything.
|
||||
- Never show empty metrics. No "Reliability: 0%" — that looks like
|
||||
the worker is unreliable. Just don't show it until there's data.
|
||||
- Lead with what the staffer KNOWS: "you placed this worker at
|
||||
Company X last month" — that's information they trust because they
|
||||
lived it.
|
||||
|
||||
## What This Means for the Architecture
|
||||
|
||||
- The vector index works on whatever text exists. A name + role is
|
||||
200 characters. That's enough for an embedding. As more data
|
||||
arrives, the embeddings get richer and search gets better.
|
||||
- The hybrid search works with sparse SQL filters too. "role =
|
||||
'Forklift Operator'" is a valid filter even without reliability.
|
||||
- The playbook system captures the staffer's decisions: "you picked
|
||||
this worker for this contract" → that IS the training data for
|
||||
future AI recommendations.
|
||||
|
||||
## Implementation
|
||||
|
||||
1. Sparse profile ingest: accept CSV with as few as 2 columns
|
||||
(name, phone). Everything else optional.
|
||||
2. Graceful UI degradation: worker cards only show fields that exist
|
||||
3. Progressive enrichment: timesheet ingest → auto-calculate
|
||||
reliability; check-in ingest → auto-calculate availability
|
||||
4. Trust indicators: "based on 3 placements" not "Reliability: 87%"
|
||||
— show WHERE the number comes from
|
||||
@@ -0,0 +1,457 @@
|
||||
# PRD — Universal AI Control Plane
|
||||
|
||||
**Status:** Long-horizon architecture target as of 2026-04-22. Lakehouse Phases 0-37 (`docs/PRD.md`) are preserved as the reference implementation and first domain-specific consumer. Phases 38+ (control-plane layers) are sequenced below.
|
||||
|
||||
**Current domain: staffing.** The immediate proving ground is the staffing substrate already built — synthetic workers_500k, contracts, emails, SMS drafts, playbook memory. Everything Phase 38-44 ships is validated first against that domain. The DevOps / Terraform / Ansible framing from the original PRD draft stays as a **long-horizon target** — architecture-compatible but not in current scope. See §Long-horizon domains at the bottom.
|
||||
|
||||
**Owner:** J
|
||||
|
||||
**Cross-read:** `docs/PRD.md` for what's shipped (staffing + AI substrate, 13 crates, ~3M rows). This doc for the layered architecture those pieces now fit into.
|
||||
|
||||
---
|
||||
|
||||
## Phase Sequencing (Phases 38-44)
|
||||
|
||||
Ship each phase before starting the next. Each ends with green tests + docs update.
|
||||
|
||||
| Phase | Layer | What ships | Est. LOC | Risk |
|
||||
|---|---|---|---|---|
|
||||
| 38 | Layer 1 skeleton | `/v1/chat`, `/v1/usage`, `/v1/sessions` routes forwarding to existing `aibridge` → Ollama. Bot migrates as first consumer. | ~400 | Low — additive, no existing routes touched |
|
||||
| 39 | Layer 3 adapters | `aibridge::ProviderAdapter` trait; Ollama + one new (OpenRouter). `/v1/chat` routes by config. | ~500 | Low-medium |
|
||||
| 40 | Layer 2 engine | Rules-based routing (`config/routing.toml`), fallback chains, cost gating. Add Gemini + Claude adapters. | ~600 | Medium |
|
||||
| 41 | Profile split | Separate Retrieval / Memory / Execution / Observer profiles; Phase 17 backward-compat. Absorbs Phase 37 hot-swap-async. | ~300 | Medium |
|
||||
| 42 | Truth Layer | New `crates/truth`; Terraform/Ansible schemas; `/v1/context` serves rules to router + observer. | ~700 | Medium |
|
||||
| 43 | Validation pipeline | Syntax/lint/dry-run/policy gates per output type. Plugs into Layer 5 execution loop. | ~400 | Medium |
|
||||
| 44 | Caller migration | All internal callers route through `/v1/chat`. Direct sidecar access deprecated. | ~200 | Low |
|
||||
|
||||
**Total ≈3100 LOC.** Phase 37 (hot-swap async) folds into Phase 41 — it's an Execution-Profile activation concern.
|
||||
|
||||
---
|
||||
|
||||
## Phase 38 — Universal API Skeleton
|
||||
|
||||
**Goal:** OpenAI-compatible `/v1/*` surface exists and forwards to existing aibridge → Ollama. Nothing about multi-provider yet — just the SHAPE, so every downstream piece (adapters, routing, usage accounting) has a surface to plug into.
|
||||
|
||||
**Ships:**
|
||||
- `crates/gateway/src/v1/mod.rs` — router + `/v1/chat`, `/v1/usage`, `/v1/sessions`
|
||||
- `crates/gateway/src/v1/ollama.rs` — shape adapter (OpenAI chat ↔ existing aibridge `GenerateRequest`)
|
||||
- One-line `nest("/v1", ...)` in `crates/gateway/src/main.rs`
|
||||
- Unit test: `POST /v1/chat` roundtrips through mocked provider
|
||||
|
||||
**Gate:**
|
||||
- `curl -X POST localhost:3100/v1/chat -d '{"model":"qwen3.5:latest","messages":[{"role":"user","content":"hi"}]}'` returns valid OpenAI-shape response.
|
||||
- `GET localhost:3100/v1/usage` returns `{requests, prompt_tokens, completion_tokens, total_tokens}`.
|
||||
- `GET localhost:3100/v1/sessions` returns `{data:[]}` (stub; real impl Phase 41).
|
||||
- `cargo test -p gateway` green.
|
||||
|
||||
**Non-goals (explicit):** streaming, tool calls, function calling, session state, multi-provider, fallback, cost gating.
|
||||
|
||||
**Risk:** Low — additive, doesn't touch existing routes. Worst case: `/v1/*` returns 502 and we fix the adapter. No existing caller affected.
|
||||
|
||||
---
|
||||
|
||||
## Phase 39 — Provider Adapter Refactor
|
||||
|
||||
**Goal:** `aibridge` grows a `ProviderAdapter` trait. Ollama implementation wraps existing sidecar code. One new provider lands as proof: **OpenRouter** (simplest — it's OpenAI-compatible, so adapter is mostly passthrough).
|
||||
|
||||
**Ships:**
|
||||
- `crates/aibridge/src/provider.rs` — `ProviderAdapter` trait with `chat()` + `embed()` + `unload()` methods
|
||||
- `crates/aibridge/src/providers/ollama.rs` — existing sidecar code moved behind the trait
|
||||
- `crates/aibridge/src/providers/openrouter.rs` — new, HTTP client to `openrouter.ai/api/v1/chat/completions`
|
||||
- `config/providers.toml` — provider registry (name, base_url, auth, default_models)
|
||||
- `/v1/chat` routes by `model` field: prefix match (e.g. `openrouter/anthropic/claude-3.5-sonnet` → OpenRouter; bare names → Ollama)
|
||||
|
||||
**Gate:**
|
||||
- `/v1/chat` with `model: "qwen3.5:latest"` hits Ollama → green
|
||||
- `/v1/chat` with `model: "openrouter/openai/gpt-4o-mini"` hits OpenRouter (key from secrets.toml) → green
|
||||
- Neither call leaks provider-specific fields upward. Response is always the `/v1/chat` shape.
|
||||
|
||||
**Non-goals:** Fallback chain (Phase 40), cost gating (Phase 40), Gemini/Claude adapters (Phase 40).
|
||||
|
||||
**Risk:** Low-medium. The trait extraction is mostly a rearrange; OpenRouter is thin. Biggest risk is secret-loading conventions — `SecretsProvider` is already in place, so reuse that path.
|
||||
|
||||
---
|
||||
|
||||
## Phase 40 — Routing & Policy Engine + Observability Recovery
|
||||
|
||||
**Goal:** Replace hardcoded T1-T5 routing with a rules engine. Add Gemini + Claude adapters. Cost gating enforced at router level. **Reinstate Langfuse + Gitea MCP** — recovery of the observability + repo-ops stack J built previously (see `project_lost_stack` memory).
|
||||
|
||||
**Ships — routing:**
|
||||
- `crates/aibridge/src/routing.rs` — rules engine (match on: task type, token budget, previous attempt failures, profile ID)
|
||||
- `config/routing.toml` — rules in TOML (human-editable, hot-reloadable)
|
||||
- `crates/aibridge/src/providers/gemini.rs` — `generativelanguage.googleapis.com` adapter
|
||||
- `crates/aibridge/src/providers/claude.rs` — `api.anthropic.com` adapter
|
||||
- Fallback chain support: if primary returns 5xx or times out, try next in chain
|
||||
- Cost gate: per-request budget + daily budget per-provider
|
||||
|
||||
**Ships — observability (was lost, now restored):**
|
||||
- **Langfuse** self-hosted via Docker Compose. Single source of truth for every LLM call trace: prompt / response / tokens / cost / latency / provider / fallback chain / profile used. UI at `localhost:3000`. Keys in `/etc/lakehouse/secrets.toml`.
|
||||
- `crates/aibridge/src/langfuse.rs` — thin fire-and-forget trace emitter. Every `/v1/chat` call spawns a background task that POSTs to `langfuse/api/public/ingestion`. Non-blocking: trace failures never affect response.
|
||||
- **Langfuse → observer pipe** — `mcp-server/langfuse_bridge.ts` or similar. Polls Langfuse's trace API at interval, forwards completed traces to observer `:3800/event` with `source: "langfuse"`. KB now sees cost/latency deltas per model, not just outcome deltas.
|
||||
- **Gitea MCP reconnect** — the MCP server binary still installed at `/home/profit/.bun/install/cache/gitea-mcp@0.0.10/` gets wired into `mcp-server/index.ts` tool registry. Agents can open PRs, comment on issues, list commits via named tools. Closes Phase 28's repo-ops gap.
|
||||
|
||||
**Gate:**
|
||||
- Rule like "local models for simple JSON emitters, cloud for reasoning" fires correctly by task type
|
||||
- Primary fails → fallback provider hits, response still matches `/v1/chat` shape
|
||||
- Daily budget hit → subsequent requests return 429 with clear retry-at header
|
||||
- `/v1/usage` reports per-provider breakdown
|
||||
- **Every `/v1/chat` call appears in Langfuse UI** with correct prompt, response, latency, token count within 2 seconds of the request completing
|
||||
- **Langfuse → observer pipe** delivers trace deltas to KB: `GET :3800/stats?source=langfuse` shows non-zero count after a few scenarios run
|
||||
- **Gitea MCP tools callable** — `list_prs`, `open_pr`, `comment_on_issue` exposed in `mcp-server/index.ts`, verifiable via a quick agent scenario
|
||||
|
||||
**Non-goals:** Retrieval Profile split (Phase 41), Truth Layer (Phase 42). Langfuse self-hosted UI customization / SSO.
|
||||
|
||||
**Risk:** Medium. Multi-provider auth + cost tracking is cross-cutting; Langfuse adds 4-5 Docker containers (PostgreSQL, ClickHouse, Redis, web, worker). Mitigation: every provider call wrapped in a single `dispatch()` function so observability flows through one point; Langfuse Docker Compose is their supported deployment path, well-tested.
|
||||
|
||||
---
|
||||
|
||||
## Phase 41 — Profile System Expansion (+ Phase 37 hot-swap async folded in)
|
||||
|
||||
**Goal:** The existing `ModelProfile` (Phase 17) becomes **ExecutionProfile**. Three new profile types land alongside. Profile activation is async — returns job_id, work runs in background (Phase 37 deliverable).
|
||||
|
||||
**Ships:**
|
||||
- `crates/shared/src/profiles/` — `ExecutionProfile`, `RetrievalProfile`, `MemoryProfile`, `ObserverProfile`
|
||||
- `crates/catalogd` gains per-profile-type CRUD endpoints (`/catalog/profiles/retrieval`, etc.)
|
||||
- `crates/vectord/src/activation.rs` — `ActivationTracker` with background-job pattern (Phase 37 content)
|
||||
- `POST /vectors/profile/{id}/activate` returns 202 + job_id, polling at `GET /vectors/profile/jobs/{id}`
|
||||
- Single-flight guard: refuse new activation if one is pending/running
|
||||
- Backward compat: `ModelProfile` still loads, aliased to ExecutionProfile
|
||||
|
||||
**Gate:**
|
||||
- Activate a profile → returns 202 in <100ms → job completes in background → `/vectors/profile/jobs/{id}` shows progress + final report
|
||||
- `tests/multi-agent/run_stress.ts` Phase 3 (hot-swap stress) passes (was SKIPPED)
|
||||
- Retrieval + Memory + Observer profiles can be created independently of Execution profile
|
||||
|
||||
**Non-goals:** Truth Layer (Phase 42), validation (Phase 43), caller migration (Phase 44).
|
||||
|
||||
**Risk:** Medium. Schema change + async refactor. Mitigation: `#[serde(default)]` on all new fields; existing profiles load unchanged.
|
||||
|
||||
---
|
||||
|
||||
## Phase 42 — Truth Layer (staffing rules first)
|
||||
|
||||
**Goal:** New `crates/truth` crate holds immutable task-class constraints. Served via `/v1/context` to router and observer. No layer can override truth. **Staffing rules ship first**; Terraform/Ansible rule shapes are scaffolded but unpopulated until the long-horizon phase.
|
||||
|
||||
**Ships:**
|
||||
- `crates/truth/src/lib.rs` — `TruthStore` with schema loading (TOML/YAML rules)
|
||||
- `crates/truth/src/staffing.rs` — staffing rule shapes:
|
||||
- Worker eligibility (active status, not blacklisted for client, geo match, role match, availability window)
|
||||
- Contract invariants (deadline present, role/count/city/state populated, budget_per_hour_max ≥ 0)
|
||||
- PII handling (redaction rules on fields tagged `PII` before any cloud call — covers existing Phase 10 sensitivity tags)
|
||||
- Client blacklist enforcement (auto-applied before any fill proposal)
|
||||
- Fill requirements (endorsed_names count matches target_count, no duplicate worker_ids within a single fill)
|
||||
- `crates/truth/src/devops.rs` — **scaffold only**: empty rule struct for Terraform/Ansible, populated in the long-horizon phase. Keeps the dispatcher signature stable so no refactor needed later.
|
||||
- `truth/` dir at repo root — rule files, versioned in git
|
||||
- `/v1/context` endpoint — returns applicable rules for a task class (`staffing.fill`, `staffing.rescue`, `staffing.sms_draft`, etc.)
|
||||
- Router consults truth before dispatching: if task violates a rule, hard-fail with structured error + rule citation (matches existing Phase 13 access-control pattern)
|
||||
|
||||
**Gate:**
|
||||
- Submit a fill proposal where a worker is client-blacklisted — router returns 422 + rule citation, no cloud tokens burned
|
||||
- Submit a fill with `endorsed_names.length != target_count` — 422 before dispatch
|
||||
- Observer cannot promote a correction that violates truth (rejected at router gate)
|
||||
- PII redaction verified: SSN / salary fields stripped from prompts before cloud calls
|
||||
- Truth reload is explicit (no file-watch hot reload in this phase)
|
||||
|
||||
**Non-goals:** Validation execution (Phase 43), policy learning / evolution (deferred), actual Terraform/Ansible rules (long-horizon phase).
|
||||
|
||||
**Risk:** Medium. Domain-specific rule enumeration takes discovery — start with a minimal rule set (5-10 staffing rules, derived from existing Phase 10-13 work) and grow organically as real fills surface edge cases.
|
||||
|
||||
---
|
||||
|
||||
## Phase 43 — Validation Pipeline (staffing outputs first)
|
||||
|
||||
**Goal:** Staffing outputs run through schema / completeness / consistency / policy gates. Plug into Layer 5 execution loop — failure triggers observer-correction iteration. This is where the **0→85% pattern reproduces on real staffing tasks** — the iteration loop with validation in place is what made small models successful.
|
||||
|
||||
**Ships:**
|
||||
- `crates/validator/src/lib.rs` — `Validator` trait: `validate(artifact) -> Result<Report, ValidationError>` + `Artifact` enum over output types
|
||||
- `crates/validator/src/staffing/fill.rs` — fill-proposal validator:
|
||||
- Schema compliance (propose_done shape matches `{fills: [{candidate_id, name}]}`)
|
||||
- Completeness (endorsed count == target_count)
|
||||
- Worker existence (every candidate_id present in workers_500k via SQL lookup)
|
||||
- Status check (every worker has status=active, not_on_client_blacklist)
|
||||
- Geo/role match (worker city/state/role matches contract)
|
||||
- `crates/validator/src/staffing/email.rs` — generated email/SMS drafts:
|
||||
- Schema (TO/BODY fields present)
|
||||
- Length (SMS ≤ 160 chars; email subject ≤ 78 chars)
|
||||
- PII absence (no SSN / salary leaked into outgoing text)
|
||||
- Worker-name consistency (name in message matches worker record)
|
||||
- `crates/validator/src/staffing/playbook.rs` — sealed playbook:
|
||||
- Operation format (`fill: Role xN in City, ST`)
|
||||
- endorsed_names non-empty, ≤ target_count × 2
|
||||
- fingerprint populated (Phase 25 validity window requirement)
|
||||
- `crates/validator/src/devops.rs` — **scaffold only**: stubbed Terraform/Ansible validators (`terraform validate`, `ansible-lint`) for the long-horizon phase
|
||||
- Task execution loop in gateway: generate → validate → if fail, observer correction + retry (bounded by `max_iterations=3`)
|
||||
- Validation results logged to observer (`data/_observer/ops.jsonl`) + KB (`data/_kb/outcomes.jsonl`)
|
||||
|
||||
**Gate:**
|
||||
- Generate a fill proposal → validator catches a phantom worker_id → observer + cloud rescue propose correction → retry → green. This reproduces the 0→85% pattern on the live staffing pipeline.
|
||||
- `/v1/usage` shows iteration count per task, provider fallback chain, and tokens-per-iteration. Cost attribution per task class visible.
|
||||
- Reproduces 14× citation-lift finding from Phase 19 refinement on similar geos after validation gates.
|
||||
|
||||
**Non-goals:** Caller migration (Phase 44), Terraform/Ansible wired validation (long-horizon).
|
||||
|
||||
**Risk:** Medium. Validation shapes have to match actual executor outputs; mitigation is using real scenario runs as test fixtures (we have ~100 of them in `tests/multi-agent/playbooks/`).
|
||||
|
||||
---
|
||||
|
||||
## Phase 44 — Caller Migration + Direct-Provider Deprecation
|
||||
|
||||
**Goal:** Every internal LLM caller routes through `/v1/chat`. Direct sidecar / direct Ollama / direct OpenAI calls are removed or explicitly deprecated with a warning.
|
||||
|
||||
**Ships:**
|
||||
- `aibridge::AiClient` becomes a thin `/v1/chat` client (was direct-to-sidecar)
|
||||
- `crates/vectord::agent` (autotune): routes through `/v1`
|
||||
- `crates/vectord::autotune`: routes through `/v1`
|
||||
- `tests/multi-agent/agent.ts::generate()`: routes through `/v1`
|
||||
- `bot/propose.ts`: routes through `/v1` (already proposed as Phase 38's test consumer, formalized here)
|
||||
- Lint rule / grep pre-commit hook: no `fetch.*:3200/generate` outside the provider adapters
|
||||
|
||||
**Gate:**
|
||||
- `grep -r "localhost:3200/generate\|/api/generate"` returns only adapter files + deprecation shims
|
||||
- `/v1/usage` accounts for every LLM call in the system within a 1-minute window after hitting a fresh scenario
|
||||
- Full scenario passes end-to-end without any caller bypassing `/v1/*`
|
||||
|
||||
**Non-goals:** New features. This phase is purely mechanical migration.
|
||||
|
||||
**Risk:** Low. Mechanical. Tests catch regressions.
|
||||
|
||||
---
|
||||
|
||||
## Phase 45 — Doc-drift detection + context7 integration
|
||||
|
||||
**Goal:** Playbooks know which external docs they were written against. When those docs change (Docker adds a feature, npm lib goes major, Terraform renames a resource), the playbook is automatically flagged. Small models never run confidently-outdated procedures — the drift signal reaches them before the next execution does.
|
||||
|
||||
**Why this phase exists at all:** The 0→85% thesis depends on the hyperfocus lane staying valid. External doc drift invalidates the lane silently — popular playbooks can compound the wrong way, accumulating boost while growing more wrong. Phase 25 already retires playbooks on *internal* schema drift; Phase 45 is the same mechanism against *external* doc drift. This is the completion of the learning loop, not an optional add-on.
|
||||
|
||||
**Ships:**
|
||||
- `shared::types::DocRef` — `{ tool: String, version_seen: String, snippet_hash: Option<String>, source_url: Option<String>, seen_at: DateTime<Utc> }`
|
||||
- `PlaybookEntry.doc_refs: Vec<DocRef>` — `#[serde(default)]` so pre-Phase-45 entries load as empty vec
|
||||
- `/vectors/playbook_memory/seed` + `/revise` accept `doc_refs` in the request body
|
||||
- `/vectors/playbook_memory/doc_drift/check/{id}` — manual drift check: looks up each `doc_refs[]` entry via the context7 bridge, returns per-tool `{version_seen, version_current, drifted: bool}` plus overall verdict
|
||||
- `/vectors/playbook_memory/doc_drift/scan` — batch scan across all active playbooks (scheduled path for Phase 45.2)
|
||||
- `mcp-server/context7_bridge.ts` — Bun HTTP bridge. Exposes `GET /docs/:tool/version` + `GET /docs/:tool/:version/diff?since=X` against the installed context7 MCP plugin. Gateway calls this over localhost.
|
||||
- `PlaybookMemory::compute_boost_for_filtered_with_role` — excludes entries where `doc_drift_flagged_at.is_some() && doc_drift_review.is_none()` (same rule as retired + superseded)
|
||||
- Overview model synthesis writes `data/_kb/doc_drift_corrections.jsonl` per detected drift: `{playbook_id, tool, version_seen, version_current, diff_summary, recommended_action, generated_at}`
|
||||
- Human-in-the-loop re-seal path: `/vectors/playbook_memory/doc_drift/resolve/{id}` — marks reviewed, optionally triggers `revise_entry` if procedure changed
|
||||
|
||||
**Gate:**
|
||||
- Seal a playbook referencing Docker 24.x → doc_refs captured. Bump Docker version behind the scenes → `/doc_drift/check/{id}` returns `drifted: true, from: 24.0.7, to: 25.0.1, summary: "..."`. The boosted playbook count on next `/vectors/hybrid` query drops by 1 (drift-flagged skipped).
|
||||
- `doc_drift_corrections.jsonl` contains the overview model's synthesis for the drift with at least: summary of change, recommended action, cost/impact estimate.
|
||||
- Human calls `/doc_drift/resolve/{id}` after reviewing → playbook returns to active boost pool (or supersedes via Phase 27 if procedure materially changed).
|
||||
- Unit tests: DocRef serde default (legacy entries load as empty), drift check against mocked context7 bridge, boost exclusion when drifted+unreviewed.
|
||||
|
||||
**Non-goals (explicit):**
|
||||
- Automatic re-seal without human review. Drift-detection → flag, not silent rewrite.
|
||||
- Cross-playbook propagation of one drift diagnosis. Each playbook reviewed individually (aggregation later if warranted).
|
||||
- Generating the updated procedure. T3 *suggests*; human or separate bot (see `bot/`) *writes*.
|
||||
|
||||
**Risk:** Medium. The context7 bridge is new infrastructure (Bun ↔ context7 MCP plugin ↔ HTTP shape for gateway consumption). Mitigation: context7 plugin is already installed; its MCP tools return structured JSON; the bridge is thin adapter code. Start with single-tool drift check (Docker) before broadening.
|
||||
|
||||
---
|
||||
|
||||
## Long-horizon domains (not in current phase sequence)
|
||||
|
||||
The architecture was drafted with DevOps execution (Terraform, Ansible) as the eventual target. **That remains aspirational, not current scope** — we don't start wiring `terraform validate` / `ansible-lint` until the staffing domain proves the six-layer architecture at scale.
|
||||
|
||||
What "proves at scale" means concretely:
|
||||
- Phases 38-44 all shipped against staffing, green tests
|
||||
- Live staffing pipeline handles **multiple concurrent contracts** with emails + SMS + indexed playbooks via `/v1/*`
|
||||
- Observed **iteration success lift** (the 0→85% pattern) reproduced on varied staffing scenarios, not just the original proof-of-concept
|
||||
- Token + cost accounting stable across provider fallback chains under real load
|
||||
- Truth Layer rules prevent real fill errors before cloud burn (not just theoretical)
|
||||
|
||||
When staffing hits that bar, the DevOps domain lights up by:
|
||||
- Populating `crates/truth/src/devops.rs` with real Terraform/Ansible rule shapes
|
||||
- Populating `crates/validator/src/devops.rs` with `terraform validate` / `ansible-lint` shell-out
|
||||
- Adding DevOps task classes to `/v1/context` rule lookup
|
||||
- No architectural changes needed — the dispatcher, router, and execution loop stay identical.
|
||||
|
||||
Other candidate long-horizon domains (same pattern):
|
||||
- Code generation tasks (validation via `cargo check` / `bun test`)
|
||||
- SQL query generation (validation via EXPLAIN + schema compliance)
|
||||
- Data pipeline definitions (validation via lineage check + schema compliance)
|
||||
|
||||
None of these are in the current roadmap. **Staffing first, production-proven, then expand.**
|
||||
|
||||
---
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
Design and implement a universal AI control-plane API that enables:
|
||||
|
||||
- **deterministic high-stakes task execution** — the immediate domain is staffing fills (contracts, workers, emails, SMS) at scale; the same architecture extends later to DevOps (Terraform, Ansible) without redesign
|
||||
- iterative capability amplification via observer loops
|
||||
- hybrid local + cloud model orchestration
|
||||
- structured knowledge + memory + playbook reuse
|
||||
- controlled improvement over time through validated iteration
|
||||
|
||||
The system prioritizes **validated pipeline success over raw model intelligence**.
|
||||
|
||||
### Current scope — staffing at scale
|
||||
|
||||
The architecture must make the already-built staffing substrate reliably answer millions of inputs: pull real data, graph it across contracts, handle multiple concurrent contracts, index emails + SMS + playbooks via the hybrid SQL+vector method, and get **faster and better each iteration** via the feedback loops (Phase 19 playbook boost, Phase 22 KB pathway recommender, Phase 24 observer, Phase 26 Mem0 upsert).
|
||||
|
||||
DevOps is an eventual domain — see §Long-horizon domains.
|
||||
|
||||
## 2. Core Objectives
|
||||
|
||||
### 2.1 Functional Goals
|
||||
|
||||
- Provide a single universal API for all AI interactions
|
||||
- Support multi-provider routing (local, flat-rate, token-based)
|
||||
- Enable iterative execution loops with observer correction
|
||||
- Store and reuse successful execution playbooks
|
||||
- Integrate: S3-based knowledge storage, LanceDB retrieval/indexing, Mem0 memory layer, MCP tool ecosystem
|
||||
|
||||
### 2.2 Non-Functional Goals
|
||||
|
||||
- Deterministic behavior under constrained execution
|
||||
- Full observability and cost accounting
|
||||
- Safe DevOps execution (no uncontrolled mutation)
|
||||
- Profile-driven routing and execution
|
||||
- Reproducibility of successful runs
|
||||
|
||||
## 3. System Architecture
|
||||
|
||||
### 3.1 Layer Overview
|
||||
|
||||
**Layer 1 — Universal API**
|
||||
|
||||
Single entry point for all applications. Endpoints:
|
||||
|
||||
- `/v1/chat`
|
||||
- `/v1/respond`
|
||||
- `/v1/tools`
|
||||
- `/v1/context`
|
||||
- `/v1/usage`
|
||||
- `/v1/sessions`
|
||||
|
||||
All programs must use this layer. No direct provider calls allowed.
|
||||
|
||||
**Layer 2 — Routing & Policy Engine**
|
||||
|
||||
Responsibilities: provider selection, fallback logic, cost gating, premium access control, profile enforcement.
|
||||
Routing based on: task type, constraints, execution profile, system health.
|
||||
|
||||
**Layer 3 — Provider Adapter Layer**
|
||||
|
||||
Normalizes all providers: Ollama (local), OpenRouter, Gemini (direct), Claude (direct or routed), future providers.
|
||||
Guarantee: no provider-specific logic leaks upward.
|
||||
|
||||
**Layer 4 — Knowledge & Memory Plane**
|
||||
|
||||
- Knowledge (S3 + LanceDB): raw documents, processed chunks, embeddings, index profiles
|
||||
- Memory (Mem0): extracted facts, entity-linked memory, session-aware retrieval
|
||||
- Playbooks: successful execution traces, reusable patterns, correction strategies
|
||||
|
||||
**Layer 5 — Execution Loop**
|
||||
|
||||
Each task runs through: Retrieval → Planning → Generation → Validation → Observer feedback → Iteration (if needed).
|
||||
|
||||
**Layer 6 — Observability & Accounting**
|
||||
|
||||
Every request logs: tokens (input/output), cost, latency, provider, fallback chain, profile used, iteration delta.
|
||||
|
||||
## 4. Execution Model
|
||||
|
||||
### 4.1 Iterative Loop
|
||||
|
||||
Each task follows: **Attempt → Validate → Observe → Adjust → Retry**
|
||||
|
||||
Constraints:
|
||||
|
||||
- max iterations (default: 3)
|
||||
- minimum improvement threshold
|
||||
- cost ceiling per task
|
||||
|
||||
### 4.2 Observer Role
|
||||
|
||||
Observer can: analyze failure, suggest corrections, recommend profile changes.
|
||||
Observer cannot: modify truth layer, auto-promote changes, override constraints.
|
||||
|
||||
### 4.3 Cloud Escalation
|
||||
|
||||
Cloud models (Gemini, Claude) are used for: structural correction, reasoning gaps, complex decomposition.
|
||||
They are not used for: brute-force retries, bulk execution.
|
||||
|
||||
## 5. Profile System
|
||||
|
||||
### 5.1 Profile Types
|
||||
|
||||
- **Retrieval Profile** — chunking strategy, embedding method, reranking rules
|
||||
- **Memory Profile** — memory weighting, context injection rules
|
||||
- **Execution Profile** — allowed providers, tool access, risk level
|
||||
- **Observer Profile** — mutation aggressiveness, iteration strategy
|
||||
|
||||
### 5.2 Profile Constraints
|
||||
|
||||
- only one major profile change per iteration
|
||||
- profiles must produce measurable deltas
|
||||
- promotion requires repeated success
|
||||
|
||||
## 6. Truth Layer (Critical)
|
||||
|
||||
Defines non-negotiable constraints:
|
||||
|
||||
- Terraform rules
|
||||
- Ansible structure requirements
|
||||
- security policies
|
||||
- organization standards
|
||||
|
||||
Rules:
|
||||
|
||||
- immutable at runtime
|
||||
- referenced by all layers
|
||||
- cannot be overridden by observer
|
||||
|
||||
## 7. Playbook System
|
||||
|
||||
### 7.1 Playbook Definition
|
||||
|
||||
Each successful run produces: task class, context used, steps executed, tools used, output artifacts, validation results, cost/latency, success score.
|
||||
|
||||
### 7.2 Playbook Lifecycle
|
||||
|
||||
- created on success
|
||||
- reused for similar tasks
|
||||
- decayed over time
|
||||
- pruned if ineffective
|
||||
|
||||
## 8. Validation System
|
||||
|
||||
All DevOps outputs must pass: syntax validation, linting, dry-run, policy compliance.
|
||||
Failure → iteration continues or task fails.
|
||||
|
||||
## 9. MCP Integration
|
||||
|
||||
MCP servers provide: tools, external data, execution capabilities.
|
||||
All MCP outputs must be: normalized, validated, schema-compliant.
|
||||
No direct MCP output reaches the model.
|
||||
|
||||
## 10. Token Accounting & Budget Control
|
||||
|
||||
Each request tracks: input tokens, output tokens, retries, fallback cost.
|
||||
Policies: premium providers gated, cost ceilings enforced, per-task budget limits.
|
||||
|
||||
## 11. Failure Handling
|
||||
|
||||
**Recoverable failures:** bad decomposition, missing steps, weak retrieval → observer + iteration.
|
||||
|
||||
**Hard failures:** missing truth data, invalid task classification, unsafe execution → termination + error report.
|
||||
|
||||
## 12. Success Criteria
|
||||
|
||||
A task is successful only if:
|
||||
|
||||
- output is valid
|
||||
- all validators pass
|
||||
- no policy violations
|
||||
- result is reproducible
|
||||
- cost within limits
|
||||
|
||||
## 13. Key Risks & Mitigations
|
||||
|
||||
- **Observer drift** → bounded authority, confidence tracking
|
||||
- **Memory poisoning** → validation layer, memory weighting
|
||||
- **Cost explosion** → token accounting, iteration caps
|
||||
- **Retrieval errors** → post-retrieval validation, profile tuning
|
||||
@@ -94,3 +94,8 @@
|
||||
**Date:** 2026-04-16
|
||||
**Decision:** Keep Parquet + binary-blob vectors + in-RAM HNSW as the primary vector backend. Add Lance as a second backend available per-profile for workloads where Lance wins architecturally. Per-profile `vector_backend: Parquet | Lance` field becomes part of Phase 17 model profiles. Implementation kicks off via the standalone `crates/lance-bench` crate and is promoted into `vectord::lance_store` when the API stabilizes.
|
||||
**Rationale:** Head-to-head benchmark on the 100K × 768d `resumes_100k_v2` index (see `docs/ADR-019-vector-storage.md` for the full scorecard). Parquet+HNSW wins current-scale search latency by 2.55× (873us vs 2229us p50). Lance wins index build time by 14× (16s vs 230s), random row access by 112× (311us vs ~35ms full-file scan), and append speed structurally (0.08s vs full Parquet rewrite). Neither strictly dominates — the dual-use PRD framing (staffing + LLM brain) means both workloads exist in the same system. Keeps ADR-008's "Parquet is the format" principle intact for dataset tables; adds Lance as a purpose-built vector-tier option without discarding the tuned HNSW stack.
|
||||
|
||||
## ADR-020: `register()` is idempotent by name with a schema-fingerprint gate
|
||||
**Date:** 2026-04-19
|
||||
**Decision:** `catalogd::Registry::register(name, fingerprint, objects)` is idempotent on `name`. If no manifest for `name` exists, create one. If one exists with the same `schema_fingerprint`, reuse its `DatasetId`, replace `objects`, bump `updated_at`, and write through. If one exists with a different `schema_fingerprint`, reject with `409 Conflict` (HTTP) / `FAILED_PRECONDITION` (gRPC). A one-shot operator endpoint `POST /catalog/dedupe` collapses any pre-existing duplicates (preferring the manifest with a non-null `row_count`, then the most recently updated).
|
||||
**Rationale:** Registry was keyed by surrogate `DatasetId` with no uniqueness constraint on `name`, so every caller that re-registered (re-ingest, external cron, gRPC retry) silently created a parallel manifest pointing at the same parquet — accumulating 308× `successful_playbooks` in live state before detection. The fingerprint gate turns re-ingest into an explicit no-op (matching PRD invariant #5 "ingestd is idempotent — re-ingesting the same file is a no-op") while forcing schema drift to be visible instead of silently clobbering. 409 status separates policy rejections from server errors, which matters for the Phase 12 tool-consumer ecosystem. Concurrency: the write lock is held across the storage write to close the check→insert TOCTOU window; serializing registers is acceptable because registers-per-second is low on the ingest path. Audit: idempotent-register events are visible as bumps to the stored manifest's `updated_at` field and in `catalogd` tracing output (tracing is non-durable, operator view only); `DedupeReport` is the return-value audit for cleanup runs. No event-journal entries are emitted — ADR-012 scopes the journal to row-level mutations, not catalog-manifest operations.
|
||||
|
||||
+89
-6
@@ -188,7 +188,7 @@
|
||||
- Endpoint: `POST /catalog/datasets/by-name/{name}/tombstone` accepting `{row_key_column, row_key_values[], actor, reason}`; companion `GET` lists active tombstones
|
||||
- `queryd::context::build_context` wraps tombstoned tables: raw goes to `__raw__{name}`, public name becomes a DataFusion view with `WHERE CAST(col AS VARCHAR) NOT IN (...)` filter
|
||||
- End-to-end on candidates: tombstone 3 IDs, COUNT drops 100,000 → 99,997, specific WHERE returns empty, AiView candidates_safe transitively excludes them too, restart preserves all tombstones
|
||||
- Limits / not in MVP: physical compaction (Phase 8 doesn't yet read tombstones during merge); journal integration (tombstones don't yet emit Phase 9 mutation events — covered by audit fields on the tombstone itself)
|
||||
- Limits / not in MVP: journal integration (tombstones don't yet emit Phase 9 mutation events — covered by audit fields on the tombstone itself). Physical compaction integration was closed by Phase E.2 below.
|
||||
- [x] Phase D: AI-safe views — 2026-04-16
|
||||
- `shared::types::AiView` — name, base_dataset, columns whitelist, optional row_filter, column_redactions
|
||||
- `shared::types::Redaction` — Null | Hash | Mask { keep_prefix, keep_suffix }
|
||||
@@ -230,7 +230,7 @@
|
||||
- Profile-driven routing: `POST /vectors/profile/{id}/search` auto-routes to Lance when profile.vector_backend=lance
|
||||
- Auto-migrate + auto-index on activation
|
||||
- Measured on real 100K × 768d: migrate 0.57s, IVF_PQ build 16.2s (14× faster than HNSW 230s), search 23ms, append 100 rows 3.3ms, doc_id fetch 3.5ms (with scalar btree)
|
||||
- IVF_PQ recall@10 = 0.805 (HNSW = 1.000) — measured via `/vectors/lance/recall/{idx}` harness
|
||||
- IVF_PQ recall@10 = 0.805 with Lance's default `nprobes=1` (the hidden cap — see 2026-04-20 tuning work below, which lifts it to 1.000). Measured via `/vectors/lance/recall/{idx}` harness.
|
||||
- [x] Phase E.3: Scheduled ingest — 2026-04-17
|
||||
- `ingestd::schedule` module: ScheduleDef, ScheduleStore (JSON at `_schedules/{id}.json`), Scheduler tokio task
|
||||
- Supports MySQL + Postgres sources on interval triggers (Cron variant defined, parsing stubbed)
|
||||
@@ -241,10 +241,93 @@
|
||||
- Two-tier: lopdf text extraction → Tesseract 5.5 fallback for scanned/image PDFs
|
||||
- Extracts embedded XObject /Image streams, shells to tesseract --oem 3 --psm 6
|
||||
- Same schema (source_file, page_number, text_content) — downstream unchanged
|
||||
- [ ] Fine-tuned domain models
|
||||
- [ ] Multi-node query distribution
|
||||
- [x] Catalog hygiene — idempotent `register()` + dedupe + DELETE (2026-04-19, ADR-020)
|
||||
- `catalogd::Registry::register` now gates on `(name, schema_fingerprint)`: same fp → reuse `DatasetId` and update objects in place; different fp → return error (409 Conflict on HTTP, `FAILED_PRECONDITION` on gRPC). First-time registration is unchanged.
|
||||
- `POST /catalog/dedupe` one-shot operator endpoint collapses pre-existing duplicates; winner = non-null `row_count` first, newest `updated_at` second.
|
||||
- `DELETE /catalog/datasets/by-name/{name}` removes the manifest from both in-memory registry and object storage (metadata-only — parquet files, vector indexes, tombstones are NOT cascade-deleted). Added to support test-harness cleanup; also plugs a real catalog hole where zombie entries from prior deletes would break DataFusion schema inference.
|
||||
- Cleanup run on live catalog: 374 → 31 datasets, 343 orphan manifests removed, 0 errors. 308× `successful_playbooks` was the worst offender.
|
||||
- Concurrency: write lock held across storage I/O in `register()` to close the check→insert TOCTOU window (32-worker multi-threaded stress test verifies single-manifest invariant).
|
||||
- End-to-end verification: `scripts/e2e_pipeline_check.sh` runs 31 assertions across 12 pipeline stages (ingest → catalog → SQL+JOIN → dedup → idempotency → metadata → PII → vector embed → semantic search → cleanup) against the live gateway. Idempotent across repeat runs.
|
||||
- Tests: 11 new in `catalogd` (was 0, includes 3 concurrency tests + 3 delete_dataset tests); 11 new in `storaged` for `AppendLog` + `ErrorJournal` (was 0). Fixed a broken doctest in `append_log.rs`.
|
||||
- [x] Autotune agent: portfolio rotation + auto-bootstrap (2026-04-20)
|
||||
- `pick_periodic_target` now sources candidates from `IndexRegistry` (not just promoted indexes) and picks least-recently-tuned, so trial budget spreads across every index with ≥1000 vectors instead of fixating on one converged champion.
|
||||
- `run_one_cycle` bootstraps on first visit: `ensure_auto_harness` auto-generates `{index}_auto` (20 synthetic self-queries, k=10, brute-force ground truth) if missing, then seeds with `HnswConfig::default()` (ec=80/es=30).
|
||||
- Regression fix: `harness::recall_at_k` now uses set-intersection semantics. The prior impl counted duplicates in `predicted` — on corpora with repeated chunks (`kb_response_cache_agent`) this inflated recall above 1.0 and poisoned promotion decisions. +7 unit tests.
|
||||
- [x] Scheduled ingest: real cron parsing (2026-04-20)
|
||||
- Vixie-compatible 5/6-field cron via `croner` crate. Day-of-week follows Unix convention (`1-5` = Mon-Fri). 6-field adds seconds granularity.
|
||||
- `validate_trigger` in `ingestd::schedule` — create/patch handlers reject malformed expressions with `400 BAD_REQUEST` at creation time, not silently at fire time.
|
||||
- Swapped away from the `cron` crate (0.16) which uses a non-Unix DOW convention (`1=Sun`) that would silently bite anyone writing `1-5` expecting weekdays. +9 unit tests.
|
||||
- [x] EvalSets federation (2026-04-20)
|
||||
- `harness::HarnessStore` mirrors the TrialJournal / PromotionRegistry federation pattern: eval artifacts colocate with each index's recorded bucket; legacy evals in primary remain discoverable via a fallback path; cross-bucket listing dedupes.
|
||||
- Every eval callsite (service.rs × 5, agent.rs × 3, autotune.rs × 1) now routes through `HarnessStore`. `VectorState` and `AgentDeps` each hold a shared instance.
|
||||
- [x] Index bucket-migrate PATCH (2026-04-20)
|
||||
- `PATCH /vectors/indexes/{name}/bucket` copies an index's vector parquet + trial-journal batches + promotion file + auto-harness to `dest_bucket`, flips `IndexMeta.bucket` as the commit point, and evicts the `EmbeddingCache` so next load reads from the new bucket. Optional `delete_source: true` sweeps source artifacts.
|
||||
- Lance-backed indexes refused with 400 — Lance URIs are bucket-specific and require rewriting the dataset, separate story. Round-trip verified: 390 artifacts, 0.04s.
|
||||
- [x] IVF_PQ recall tuning (2026-04-20)
|
||||
- `LanceVectorStore::search` now accepts optional `nprobes` + `refine_factor`. Lance's built-in `nprobes=1` default was the hidden cap on recall — on 316-partition `resumes_100k_v2` it searched only 0.3% of partitions per query.
|
||||
- Server defaults (`LANCE_DEFAULT_NPROBES=20`, `LANCE_DEFAULT_REFINE_FACTOR=5`) flow through the scoped-search path and the autotune harness. Measured on `resumes_100k_v2`: recall `0.805 → 1.000` at p50 ≈ 7.4ms. Even `nprobes=5, refine=5` saturates recall at p50 ≈ 4.7ms.
|
||||
- `/vectors/lance/recall/{idx}` accepts per-request `nprobes` / `refine_factor` so operators can sweep the curve.
|
||||
- [x] **Phase 19: Playbook memory (meta-index)** — the feedback loop originally implied by the PRD but never built. Playbooks stop being write-only; they start shaping future rankings. (2026-04-20)
|
||||
- [x] 19.1 — `POST /vectors/playbook_memory/rebuild` scans `successful_playbooks` via DataFusion, builds one `PlaybookEntry` per row (operation + approach + context embedded as one vector via nomic-embed-text)
|
||||
- [x] 19.2 — Brute-force cosine search over in-memory embeddings (chosen over HNSW: successful_playbooks maxes around thousands of rows, overhead of a second indexed surface isn't worth it until that ceiling bites)
|
||||
- [x] 19.3 — Endorsed names parsed out of `result` column, keyed by `(city, state, name)` tuple so shared names across cities don't cross-pollinate. Parsing via `parse_names` + `parse_city_state` helpers (7 unit tests)
|
||||
- [x] 19.4 — `/vectors/hybrid?use_playbook_memory=true`: fetches `top_k * 5` candidates so endorsed workers outside the vanilla top-K can still climb. Boost is additive on vector score, each hit carries `playbook_boost` + `playbook_citations` in the response for explainability
|
||||
- [x] 19.5 — Multi-agent orchestrator (`tests/multi-agent/orchestrator.ts`) auto-seeds `POST /vectors/playbook_memory/seed` on consensus_done, so the next query sees the new endorsement without a full `/rebuild`. Closes the feedback loop: two agents reach consensus → playbook sealed → next query re-ranks
|
||||
- [x] 19.6 — `MAX_BOOST_PER_WORKER = 0.25` enforced in `compute_boost_for`; verified with unit test (100 identical playbooks → boost capped at 0.25) and live test (5 identical seeds → exactly 0.25). Time decay also wired: `BOOST_HALF_LIFE_DAYS = 30.0` — 30-day-old playbooks contribute half, 60-day a quarter, via `exp(-age_days / 30)` in the boost loop
|
||||
- Real finding surfaced during build: the 32 bootstrap rows in `successful_playbooks` reference phantom worker names — 80 of 82 don't correspond to actual rows in `workers_500k`. `/seed` endpoint bypasses `successful_playbooks` so operators can prime memory with real fixtures; production path is the orchestrator write-through
|
||||
- [x] **Phase 19 refinement — geo + role prefilter on boost** (2026-04-21)
|
||||
- Added `compute_boost_for_filtered` and `compute_boost_for_filtered_with_role` to `playbook_memory.rs`. SQL filter's `(city, state, role)` parsed in `service.rs`; exact role-matches in target geo skip cosine and earn similarity=1.0. Restored the feedback loop: matched=0 → matched=11 per query on the same Nashville test. Citation density on Riverfront Steel: 2 → 28 per run (14×).
|
||||
- Rust unit tests: `extractor_tests::extract_target_geo_basic/_missing_state/_word_boundary`, `extract_target_role_basic/_none/_multi_word`. 6/6 pass.
|
||||
- Diagnostic log: `playbook_boost: boosts=N sources=N parsed=N matched=N target_geo=? target_role=?` on every call.
|
||||
- [x] **Phase 20: Model Matrix + Overseer Tiers** (2026-04-21)
|
||||
- `config/models.json` — 5 tiers (t1_hot / t2_review / t3_overview / t4_strategic / t5_gatekeeper), each with context_window + context_budget + overflow_policy. Ollama Cloud bearer key from `/root/llm_team_config.json`.
|
||||
- Hot path: qwen3.5:latest + qwen3:latest local with `think:false`. Mistral dropped after 0/14 fill on complex scenarios.
|
||||
- T3 cloud: gpt-oss:120b via Ollama Cloud — verified 4-8s latency, strict JSON-shape output for remediation.
|
||||
- [x] **Phase 21: Scratchpad + Tree-Split Continuation** (2026-04-21)
|
||||
- `tests/multi-agent/agent.ts`: `estimateTokens()`, `assertContextBudget()`, `generateContinuable()`, `generateTreeSplit()`. `think` flag plumbed through sidecar's `/generate`. Empty-response backoff + truncation-continuation, no max_tokens tourniquet.
|
||||
- Rust port shipped (2026-04-21, companion to Phase 27):
|
||||
- `crates/aibridge/src/context.rs` — `estimate_tokens` (chars/4 ceil, matches TS), `context_window_for`, `assert_context_budget` returning `Result<BudgetCheck, (BudgetCheck, usize over_by)>` so callers get the numbers back on both success and overflow. Windows table mirrors `config/models.json`.
|
||||
- `crates/aibridge/src/continuation.rs` — `generate_continuable<G: TextGenerator>` handles the two failure modes from TS: (a) empty thinking-model response → geometric-backoff retry with 2× budget up to `budget_cap`; (b) truncated non-empty → continuation with partial-as-scratchpad. `is_structurally_complete` balances braces then JSON.parse-check for the JSON shape; text shape is "non-empty". Guards the degen case "all retries empty → bail, don't loop on empty partial" — the TS impl has this implicit, Rust makes it explicit.
|
||||
- `crates/aibridge/src/tree_split.rs` — `generate_tree_split` map→reduce with running scratchpad. Per-shard + reduce-prompt budget checked through `assert_context_budget`; loud-fails with the overflow message rather than silently truncating. Scratchpad oldest-digest-first truncation once it exceeds `scratchpad_budget` (default 6000 tokens, matches TS).
|
||||
- `TextGenerator` trait (native async-fn-in-trait, edition 2024) so `ScriptedGenerator` test double can inject canned sequences without a live Ollama. `AiClient` implements `TextGenerator`.
|
||||
- `GenerateRequest` gained `think: Option<bool>` field — forwards to sidecar for per-call hidden-reasoning opt-out on hot-path JSON emitters.
|
||||
- 25 aibridge unit tests (8 context + 10 continuation + 7 tree_split) — all green, no network required.
|
||||
- [x] **Phase 22: Internal Knowledge Library** (2026-04-21)
|
||||
- `data/_kb/` — signatures.jsonl, outcomes.jsonl, pathway_recommendations.jsonl, error_corrections.jsonl, config_snapshots.jsonl. Event-driven cycle: indexRun → recommendFor → loadRecommendation.
|
||||
- Item B cloud rescue: failed event → cloud remediation JSON → retry with pivot. Verified 1/3 rescues succeeded on stress_01 (Gary IN → South Bend IN pivot).
|
||||
- `scripts/kb_measure.py` aggregator. Unit tests: `kb.test.ts` — 4/4 pass (signature stability, role/city/count invariants, digest shape).
|
||||
- [x] **Phase 23: Staffer identity + competence-weighted retrieval** (2026-04-21)
|
||||
- ScenarioSpec gained `contract: ContractTerms` and `staffer: Staffer { id, name, tenure_months, role, tool_level }`.
|
||||
- tool_level runtime overrides: full / local / basic / minimal. Basic + minimal route executor to Ollama Cloud `kimi-k2.5` (kimi-k2.6 pending pro-tier upgrade).
|
||||
- `data/_kb/staffers.jsonl` — competence_score = 0.45·fill + 0.20·turn_eff + 0.20·cite + 0.15·rescue. Recomputed per run.
|
||||
- `findNeighbors` now returns `weighted_score = cosine × max_staffer_competence`. `scripts/kb_staffer_report.py` — leaderboard + cross-staffer worker overlap (Rachel D. Lewis 12× across 4 staffers → auto-discovered high-value label).
|
||||
- `gen_staffer_demo.ts` + `run_staffer_demo.sh` — 4 personas × 3 contracts = 12 runs.
|
||||
- [x] **Phase 27: Playbook versioning** (2026-04-21)
|
||||
- `PlaybookEntry` gained `version: u32` (default 1), `parent_id`, `superseded_at`, `superseded_by` fields. All `#[serde(default)]` so entries persisted before Phase 27 load as roots with version=1.
|
||||
- `PlaybookMemory::revise_entry(parent_id, new_entry)` appends a new version, stamps `superseded_at`+`superseded_by` on the parent, inherits `parent_id` and sets `version = parent + 1` on the new entry. Rejects revising a retired or already-superseded parent with a clear error — the tip of the chain is the only valid revise target.
|
||||
- `PlaybookMemory::history(playbook_id)` returns the full chain root→tip, walking `parent_id` backward then `superseded_by` forward. Cycle-safe. Works from any node in the chain.
|
||||
- Superseded entries excluded from boost (same rule as retired): `compute_boost_for_filtered_with_role`, the active-entries prefilter, the geo-index rebuild, and the upsert existing-entry search all skip `superseded_at.is_some()`.
|
||||
- Endpoints: `POST /vectors/playbook_memory/revise` + `GET /vectors/playbook_memory/history/{id}`.
|
||||
- `status_counts` now returns `(total, retired, superseded, failures)`. `/status` JSON reports `superseded` as a distinct counter; `active = total - retired - superseded`.
|
||||
- 8 unit tests under `mod version_tests` covering: chain-metadata stamping, retired-parent rejection, already-superseded-parent rejection, superseded endorsement exclusion from boost, history traversal from root/tip/middle, empty-for-unknown-id, superseded-status-count, legacy-entry-default-version round-trip. 26/26 playbook_memory tests pass.
|
||||
- [x] **Phase 24: Observer / Autotune integration** (2026-04-20, commit b95dd86)
|
||||
- Closed the gap where `lakehouse-observer.service` wrapped MCP :3700 while `tests/multi-agent/scenario.ts` hit gateway :3100 directly — observer sat idle at 0 ops across 3600+ cycles.
|
||||
- `observer.ts` gained a Bun HTTP listener on `OBSERVER_PORT` (default 3800) with `GET /health`, `GET /stats` (totals + by_source + recent scenario digest), and `POST /event` for scenario outcomes. Body shapes into `ObservedOp` with `source="scenario"` + `staffer_id` + `sig_hash` + `event_kind` + geo + rescue flags.
|
||||
- `recordExternalOp()` shared ring-buffer insert — ERROR_ANALYZER and PLAYBOOK_BUILDER loops now see both MCP-wrapped and scenario-posted ops through the same path.
|
||||
- `persistOp()` swap: old path wrote via `/ingest/file?name=observed_operations` which has REPLACE semantics (wiped prior ops); now uses append-friendly Parquet write-through.
|
||||
- [x] **Phase 25: Validity windows + playbook retirement** (2026-04-21, commit e0a843d)
|
||||
- `PlaybookEntry` gained four optional fields (`#[serde(default)]` so legacy entries load as never-expiring): `schema_fingerprint` (SHA-256 over target dataset columns at seed time), `valid_until` (RFC3339 hard expiry), `retired_at` (set by retire calls), `retirement_reason` (human string).
|
||||
- `compute_boost_for_filtered_with_role` now skips retired + expired entries before geo/cosine — no silent boosting from stale playbooks. Unit-tested on expired `valid_until` + retired + schema-drift retirement.
|
||||
- Two retirement paths: `retire_one(playbook_id, reason)` for manual, `retire_on_schema_drift(city, state, current_fingerprint, reason)` for batch schema-migration sweep. Legacy entries without a fingerprint skip drift retirement (safe).
|
||||
- Endpoint: `POST /vectors/playbook_memory/retire` — accepts either `{playbook_id, reason}` or `{city, state, current_schema_fingerprint, reason}`.
|
||||
- [x] **Phase 26: Mem0 upsert + Letta geo hot cache** (2026-04-21, commit 640db8c)
|
||||
- Mem0-style upsert: `/seed` with `append=true` (default) routes through `upsert_entry`, which decides ADD / UPDATE / NOOP on (operation, day, city, state). Same-day re-seed merges names (union, stable order) instead of duplicating the row. Identical re-seed is a no-op. Different-day same-operation is a fresh ADD. Playbook_id stays stable on UPDATE so prior citations remain valid.
|
||||
- Letta-style hot cache: `PlaybookMemory` now holds a `geo_index: HashMap<(city_lower, state_upper), Vec<entry_idx>>` rebuilt on every mutation. Geo-filtered boost queries skip the full scan and hit the O(1) key lookup. At 1.9K entries the full scan was sub-ms; the index scales the same path to 100K+ without code changes.
|
||||
- `UpsertOutcome` enum reported back to callers — `{mode: added|updated|noop, playbook_id, merged_names?}` + `entries_after`.
|
||||
- [ ] Fine-tuned domain models (Phase 25+)
|
||||
- [ ] Multi-node query distribution (only if ceilings bite)
|
||||
|
||||
---
|
||||
|
||||
**52+ unit tests | 13 crates | 19 ADRs | 2.47M rows | 100K vectors | Hybrid Parquet+HNSW ⊕ Lance**
|
||||
**Latest: 2026-04-17 — 8 commits shipping Phase 16.2 through Phase 18**
|
||||
**145 unit tests | 13 crates | 21 ADRs | 2.47M rows | 100K vectors | Hybrid Parquet+HNSW ⊕ Lance | Phases 0-27 shipped**
|
||||
**Latest: 2026-04-21 — Phase 27 (playbook versioning: `version` + `parent_id` + `superseded_at` + `superseded_by` on `PlaybookEntry`, `/revise` + `/history` endpoints, 8 new tests). Doc-sync pass: Phase 24 observer + Phase 25 validity windows + Phase 26 Mem0/Letta now reflected in phase tracker. Phase 19.6 time decay noted as wired (was misdocumented as deferred). Phase E.2 tombstone-at-compaction noted as closed in Phase 8 MVP limits.**
|
||||
|
||||
+384
-6
@@ -1,8 +1,28 @@
|
||||
# PRD: Lakehouse — Rust-First Substrate for Versioned Knowledge Stores
|
||||
|
||||
**Status:** Active — Phases 0-18 shipped; hybrid Parquet+HNSW ⊕ Lance operational; scheduled ingest live; PDF OCR live; entering horizon items
|
||||
> ## ▶ Direction pivot — 2026-04-22
|
||||
>
|
||||
> **`docs/CONTROL_PLANE_PRD.md` is now the long-horizon architecture target.**
|
||||
> Lakehouse (this document) is being refactored to conform to the Universal AI
|
||||
> Control Plane architecture — six layers, `/v1/*` universal API, multi-provider
|
||||
> routing, Truth Layer for DevOps constraints.
|
||||
>
|
||||
> This PRD remains the source of truth for **everything Phases 0-37** — the
|
||||
> shipped staffing / AI-substrate system below is preserved as the reference
|
||||
> implementation and the first domain-specific consumer of the control plane.
|
||||
> All existing phases, ADRs, and invariants continue to apply.
|
||||
>
|
||||
> **Phases 38+ live in `CONTROL_PLANE_PRD.md`** (Universal API, provider
|
||||
> adapters, routing engine, expanded profile system, Truth Layer, validation
|
||||
> pipeline, caller migration).
|
||||
>
|
||||
> Cross-read: this PRD for what's shipped, CONTROL_PLANE_PRD for where it's going.
|
||||
|
||||
---
|
||||
|
||||
**Status:** Active — Phases 0-36 shipped, Phase 37 (hot-swap async) TODO; control-plane pivot in flight; E2E test 9.0/10; stress test passes
|
||||
**Created:** 2026-03-27
|
||||
**Last reframed:** 2026-04-17 — hybrid architecture proven end-to-end on 100K vectors (see §Phase 18 + ADR-019)
|
||||
**Last updated:** 2026-04-22 — Control-plane pivot; Gateway seed concurrency (Semaphore); stress test with 6 diverse tasks
|
||||
**Owner:** J
|
||||
|
||||
---
|
||||
@@ -80,6 +100,9 @@ No new frameworks without documented ADR.
|
||||
| **aibridge** | Rust↔Python boundary — HTTP client to FastAPI sidecar |
|
||||
| **ui** | Dioxus frontend — Ask, Explore, SQL, System tabs |
|
||||
| **shared** | Types, errors, Arrow helpers, config, protobuf definitions, **secrets provider trait**, **PII detection** |
|
||||
| **mcp-server** | Agent gateway (Bun) — MCP tools, intelligence endpoints, scenario observer (:3700) |
|
||||
| **observer** | Autonomous iteration loop — records ops, error analysis, playbook consolidation (:3800) |
|
||||
| **scenario** | Day-in-the-life orchestrator — T5 tiers, KB integration, continuation tree-split (:bun) |
|
||||
|
||||
**Federation building blocks** (shipped 2026-04-16):
|
||||
- `shared::secrets::SecretsProvider` trait + `FileSecretsProvider` reading `/etc/lakehouse/secrets.toml` (0600 enforced)
|
||||
@@ -110,8 +133,8 @@ User question → gateway
|
||||
**Semantic (RAG):** "Find candidates who could do data engineering work"
|
||||
→ Embed question → vector search across resume embeddings → retrieve top chunks → LLM answers
|
||||
|
||||
**Hybrid:** "Which clients are we losing money on, and why?"
|
||||
→ SQL for margin calculations + RAG over client notes/emails for context → LLM synthesizes
|
||||
**Hybrid (shipped 2026-04-17):** "Find reliable forklift operators in Illinois with OSHA certs"
|
||||
→ `POST /vectors/hybrid` with `sql_filter` + `question`: SQL narrows to structurally-valid candidates (role, state, reliability, certs), brute-force cosine ranks by semantic relevance within the filtered set, LLM generates answer from SQL-verified records only. Zero hallucinations on the staffing simulation (16/16 positions filled, all workers verified against golden data).
|
||||
|
||||
### Invariants
|
||||
|
||||
@@ -126,6 +149,15 @@ User question → gateway
|
||||
9. **Every reader gets its own profile.** Human operators, AI agents, and local models are all clients of the same substrate. Each has a named profile with its own bucket, vector indexes, trial history, and dataset bindings. Profiles are a first-class architectural concept, not a tenancy afterthought. (Phase 17)
|
||||
10. **Trials are data, not logs.** Every index build is a trial with measurable metrics. The trial journal IS the agent's memory for how to tune itself. Stored as write-once batched JSONL per the ADR-018 append-log pattern.
|
||||
11. **Operational failures are findable in one HTTP call.** The bucket error journal, trial journal, and audit log all expose `/storage/errors`, `/hnsw/trials`, `/access/audit` with structured filter + aggregation. No `grep` archaeology to answer "what broke?"
|
||||
12. **Playbooks feed the index, not just the log.** A completed playbook isn't just a record of what worked — it's a signal that shapes future rankings. Every `successful_playbooks` row contributes to the playbook-memory vector index, so semantically-similar future operations re-rank toward workers that have actually succeeded in comparable fills. This is the "system gets smarter over time" dimension that distinguishes this substrate from a static search engine. (Phase 19)
|
||||
|
||||
---
|
||||
|
||||
## Vision drift acknowledged (2026-04-20)
|
||||
|
||||
The system as shipped through Phase 18 is a **hybrid SQL+vector search engine with a playbook log**. The original pitch (and the "staffing AI co-pilot" framing) implied a **meta-index that learns from playbooks over time** — hot-swap profiles weren't just routing, they were knowledge generations that compounded. That learning loop was never built; playbooks were write-only. Phase 19 closes that gap explicitly.
|
||||
|
||||
The feedback signal is **statistical + semantic**, not neural. No model training — the index reads the playbook journal, computes operation-similarity, and boosts endorsed workers at query time. Rebuildable from `successful_playbooks` alone, same as every other derived index.
|
||||
|
||||
---
|
||||
|
||||
@@ -349,11 +381,357 @@ The question raised 2026-04-16 after J's LLMS3 knowledge base identified Lance a
|
||||
|
||||
Per-profile `vector_backend: Parquet | Lance` becomes part of Phase 17 (model profiles). See ADR-019 for the full scorecard and caveats.
|
||||
|
||||
### Phase 19+: Further horizon
|
||||
### Phase 19: Playbook memory (meta-index) — the feedback loop
|
||||
|
||||
Make successful playbooks actually improve future searches. Today `successful_playbooks` is a write-only log; future-you looks at it and thinks "cool, we filled Toledo welders once" — but the index has no idea it happened, so the next Toledo-welder search ranks the same as if none of those fills had existed. Phase 19 closes the loop.
|
||||
|
||||
| Step | Deliverable | Gate |
|
||||
|---|---|---|
|
||||
| 19.1 | Embed every `successful_playbooks` row — operation + approach + context → one chunk per playbook | A new dataset `playbook_memory` appears in catalog with N rows = row count of `successful_playbooks` |
|
||||
| 19.2 | Vector index on `playbook_memory` (HNSW or Lance — whichever `agent-parquet` profile uses) | `/vectors/search` against `playbook_memory` returns semantically similar past playbooks |
|
||||
| 19.3 | Endorsement extraction: each playbook row has `fills[]` (worker_ids it succeeded with). Parse them out at ingest time and store in a sidecar `playbook_endorsements` Parquet keyed by playbook_id | `SELECT * FROM playbook_endorsements WHERE playbook_id = 'X'` returns the worker_ids |
|
||||
| 19.4 | `/vectors/hybrid` gains opt-in `use_playbook_memory: bool`. When true: after hybrid ranks candidates, find top-K similar past playbooks (semantic search over `playbook_memory`), extract endorsed worker_ids, add a bounded boost to candidates in the endorsed set, re-rank | A search where the "right" worker is known from a prior playbook ranks higher with the flag than without |
|
||||
| 19.5 | Write-through from multi-agent orchestrator: when two agents seal a playbook, it appends to `successful_playbooks` AND triggers a refresh of `playbook_memory` (via existing Phase C stale-mark path). Next query sees the new signal. | Run the orchestrator → inspect `playbook_memory` → see a new row. Run the same query before/after → ranking differs. |
|
||||
| 19.6 | Ceiling-aware boost: cap the per-worker boost so one popular worker can't dominate future searches. Boost decays with time (optional) so stale playbooks matter less. | Synthetic test: 100 playbooks all filled with the same worker_id; the 101st search still returns a mix, not just that one worker |
|
||||
|
||||
**Gate:** Run a real search before and after a new successful playbook lands. The endorsed workers from similar past operations rank higher in the second call. Demonstrable with a diff of the two result sets.
|
||||
|
||||
**Why this is the right version of "meta-index":** The alternative — training a neural re-ranker on (query, candidate, outcome) triples — is a weeks-long ML story and requires labeled outcome data we don't really have. The statistical-semantic version here is rebuildable from the existing playbook journal, explainable ("boosted because of similar playbooks X, Y, Z"), and invalidatable (delete a playbook → boost goes away on next rebuild). It gets 80% of the payoff at 10% of the cost. Neural re-ranking stays as a future option if it bites.
|
||||
|
||||
**Non-goals for this phase:**
|
||||
- Neural training / fine-tuning. Statistical feedback only.
|
||||
- Hard guarantees about recall lift magnitude. "Measurably better on the demo query" is the gate, not a universal quality claim.
|
||||
- Real-time recomputation on every playbook. Batched refresh via the existing stale-marking path is sufficient.
|
||||
|
||||
### Phase 19 refinement (WIRED 2026-04-21): geo-filter + role prefilter on boost
|
||||
|
||||
Item-3 diagnostic pass surfaced that `compute_boost_for` was ranking playbooks globally by cosine similarity, while candidates came from an SQL-filtered city. Result: boost map had 170 endorsed workers, 0 intersected the 50 Nashville-filtered candidates. Zero citations where there should have been dozens.
|
||||
|
||||
Fix — in `crates/vectord/src/playbook_memory.rs`:
|
||||
- `compute_boost_for_filtered(target_geo)` — skip playbooks from other cities before cosine sort.
|
||||
- `compute_boost_for_filtered_with_role(target_geo, target_role)` — multi-strategy: exact (role, city, state) match earns similarity=1.0 and fills up to half the top_k; cosine fallback fills the rest. Mirrors Mem0/Zep 2026 guidance on parallel-strategy rerank.
|
||||
|
||||
In `crates/vectord/src/service.rs`:
|
||||
- `extract_target_geo` and `extract_target_role` pull both from the executor's SQL filter.
|
||||
- `tracing::info!` emits `playbook_boost: boosts=N sources=N parsed=N matched=N target_geo=? target_role=?` on every hybrid_search. Silent-truncation class of bug now visible.
|
||||
|
||||
Citation lift measured: avg citations per run 0.32 → 1.38 after geo filter; then 2 → 28 in the single-scenario Riverfront Steel re-run after role prefilter landed. 14× delta on same scenario.
|
||||
|
||||
Unit tests: `extract_target_geo_basic`, `_missing_state_returns_none`, `_word_boundary` (rejects "civilian" substring), `extract_target_role_basic`, `_none_when_absent`, `_multi_word` — all pass (`cargo test -p vectord --lib extractor_tests`).
|
||||
|
||||
### Phase 20: Model Matrix + Overseer Tiers (WIRED 2026-04-21)
|
||||
|
||||
Five-tier routing declared in `config/models.json`. Hot path (T1/T2) stays local (qwen3.5 + qwen3 after mistral was dropped for 0/14 fill rate on complex scenarios). Cloud for overview (T3 gpt-oss:120b), strategic (T4 qwen3.5:397b), and gatekeeper (T5 kimi-k2-thinking). Every tier declares `context_window` + `context_budget` + `overflow_policy`.
|
||||
|
||||
- T1 hot: 50-200 calls/scenario, local only — `qwen3.5:latest` executor, `think:false`
|
||||
- T2 review: 5-14 calls/event, local only — `qwen3:latest` reviewer, `think:false`
|
||||
- T3 overview: 1-3 calls/scenario, cloud primary — `gpt-oss:120b` on Ollama Cloud, thinking on
|
||||
- T4 strategic: 1-10 calls/day, cloud primary
|
||||
- T5 gatekeeper: 1-5 calls/day, audit-logged
|
||||
|
||||
T3 checkpoints + cross-day lessons wired. Lessons archive to `data/_playbook_lessons/` and load back at next scenario start as `prior_lessons` in executor context. Cloud passthrough verified on stress_01 scenario with `LH_OVERVIEW_CLOUD=1` — `gpt-oss:120b` response latency consistently 4-8s, diagnosing city-pivot ("Gary IN → Chicago IL, 40mi") when target city has zero supply.
|
||||
|
||||
`think:false` is the key mechanical finding — qwen3.5 burns ~650 tokens of hidden reasoning before emitting response; hot-path JSON emitters MUST disable thinking or continuation has to paper over empty returns. T3/T4 overseers KEEP thinking (that's the point).
|
||||
|
||||
**Kimi-k2.6 upgrade path:** Current Ollama Cloud key returns 403 on kimi-k2.6 (`ollama run kimi-k2.6:cloud` requires `ollama signin` with pro-tier account). kimi-k2.5 substitutes on the current tier — same family, strong at tool calling. Swap to k2.6 is a one-line change in `applyToolLevel` once the subscription lands.
|
||||
|
||||
### Phase 21: Scratchpad + Tree-Split Continuation
|
||||
|
||||
**Why this is a phase and not an optimization:** bumping `max_tokens` until a response stops truncating is a tourniquet — J called this out explicitly. As playbooks accumulate into the hundreds and responses grow, eventually SOME request will exceed SOME model's window, and we can't solve it by raising a number. The stable answer is two primitives that let us handle arbitrary-size work without losing context: a scratchpad that glues multi-call responses together, and a tree split that shards oversized inputs and reduces them back.
|
||||
|
||||
**Two primitives (WIRED 2026-04-21 in `tests/multi-agent/agent.ts`):**
|
||||
|
||||
1. **`generateContinuable()`** — handles OUTPUT overflow. Calls the model; checks structural completeness (for JSON: matched braces + JSON.parse success; for text: non-empty). If incomplete, calls again with "continue from here" + the partial response as scratchpad. Up to `max_continuations` times. No `max_tokens` tuning needed — if thinking ate the initial budget, continuation picks up the slack.
|
||||
|
||||
2. **`generateTreeSplit()`** — handles INPUT overflow. Caller passes an array of shards (semantic chunks of the corpus). For each shard: map call with running scratchpad digest. Final reduce call produces the answer. Scratchpad truncates oldest content if it approaches its own budget. If a single shard still overflows, `assertContextBudget` throws — caller must re-shard at finer granularity, NOT silently truncate.
|
||||
|
||||
**Guarantees:**
|
||||
1. No agent call can silently truncate. Either it completes, continues, or throws with numbers.
|
||||
2. No corpus is too big — `generateTreeSplit` handles any size the caller can shard.
|
||||
3. Scratchpad is the glue between multi-call responses; context is never lost, only compacted.
|
||||
4. Token estimation uses `chars / 4` (biased safe ~15%) until we wire the provider's tokenizer.
|
||||
|
||||
**What lives where now:**
|
||||
- `agent.ts::estimateTokens()` + `assertContextBudget()` + `generateContinuable()` + `generateTreeSplit()` — WIRED
|
||||
- `scenario.ts` executor + reviewer + overviewGenerate calls — migrated to `generateContinuable`
|
||||
- `config/models.json` — context_window + context_budget + overflow_policies per tier (declarative)
|
||||
|
||||
**Next sprint (Rust side, so gateway tools share it):**
|
||||
- `crates/aibridge/src/continuation.rs` — port of `generateContinuable`
|
||||
- `crates/aibridge/src/tree_split.rs` — port of `generateTreeSplit`
|
||||
- `crates/storaged/src/chunk_cache.rs` — precomputed shards keyed by corpus hash (avoid re-chunking on every T4 run)
|
||||
- `/metrics` counter: `context_continuations_total{model,shape,succeeded}`
|
||||
|
||||
**Status:** TS primitives WIRED. Rust port pending. The escalation path (tree split → bigger-context cloud model → kimi-k2:1t's 1M window → split decision into sub-decisions) is declared in `config/models.json` under `context_management.overflow_policies`.
|
||||
|
||||
### Phase 21 status update (WIRED 2026-04-21 evening)
|
||||
|
||||
Additional primitives landed after the initial commit:
|
||||
|
||||
- **`think: boolean`** flag plumbed through `generate()`, `generateCloud()`, `generateContinuable()`, and into sidecar's `/generate` endpoint. Enables per-call opt-out of hidden reasoning for hot-path JSON emitters. Verified: qwen3.5 with `think:false` + `num_predict:400` returns clean `{"worker_id":...}` on first call; without `think:false`, 650 tokens eaten by reasoning, response empty.
|
||||
|
||||
- **Cloud executor routing** — `ACTIVE_EXECUTOR_CLOUD` / `ACTIVE_REVIEWER_CLOUD` flags let per-staffer tool_level route executor to Ollama Cloud when weak local model (qwen2.5) would collapse. Verified on kimi-k2.5 via Ollama Cloud: clean JSON emission, think:false honored.
|
||||
|
||||
Rust port of continuation + tree-split primitives remains queued for next sprint (`crates/aibridge/src/continuation.rs`, `tree_split.rs`).
|
||||
|
||||
### Phase 22: Internal Knowledge Library (KB)
|
||||
|
||||
Meta-layer over Phase 19 playbook_memory. Playbook memory answers "which WORKERS worked for this event." The KB answers "which CONFIG worked for this playbook signature." Subject changes from workers to the system itself — model choice, budget hints, overflow policies, pathway notes.
|
||||
|
||||
**Files (`data/_kb/`):**
|
||||
- `signatures.jsonl` — (sig_hash, embedding[], first_seen, last_seen, run_count). Sig = stable hash of the sequence of (kind, role, count, city, state) across events.
|
||||
- `outcomes.jsonl` — per-run record: {sig, run_id, models, ok/total, turns, citations, per-event summary, elapsed}.
|
||||
- `pathway_recommendations.jsonl` — AI-synthesized for next run: {confidence, rationale, top_models, budget_hints, pathway_notes, neighbors_consulted}.
|
||||
- `error_corrections.jsonl` — detected fail→succeed pairs on same sig, diff of what changed.
|
||||
- `config_snapshots.jsonl` — history of models.json changes + why.
|
||||
|
||||
**Cycle (event-driven, not wall-clock):**
|
||||
1. Scenario ends → `kb.indexRun()` extracts signature, embeds spec digest, appends outcome.
|
||||
2. `kb.recommendFor(nextSpec)` finds k-NN signatures via cosine, feeds their outcome history + recent error corrections to the overview model, writes a structured recommendation.
|
||||
3. Next scenario starts → `kb.loadRecommendation(spec)` pulls the newest rec for this sig, injects `pathway_notes` into `guidanceFor()` alongside prior lessons.
|
||||
|
||||
**Why file-based for MVP:** Phase 19 playbook_memory is already a catalogd dataset. KB is a separate meta-layer; keep it file-based first to iterate without a gateway schema migration. Rust port (and promotion to vectord-indexed corpus for neighbor search at scale) lands once shape stabilizes — mirrors how Phase 21 primitives were TS-first → Rust next sprint.
|
||||
|
||||
**What the overview model gets asked:**
|
||||
- Target scenario digest
|
||||
- Top-k neighbor signatures with avg ok rate, best model combo per neighbor
|
||||
- Recent error corrections (sig, before/after model set)
|
||||
|
||||
**What it outputs (JSON-constrained):**
|
||||
- confidence (high/medium/low)
|
||||
- rationale (2-3 sentences)
|
||||
- top_models {executor, reviewer, overview}
|
||||
- budget_hints {executor_max_tokens, reviewer_max_tokens, executor_think}
|
||||
- pathway_notes (concrete pre-run advice)
|
||||
|
||||
**Status (WIRED 2026-04-21):** `tests/multi-agent/kb.ts` holds all primitives. scenario.ts reads rec at start, indexes + recommends at end. Cold start gracefully writes a "low confidence, no history" rec so the second run has a floor to build on. `snapshotConfig()` wired to fire at every scenario start — active model set + tool_level + cloud flags hashed and appended to `config_snapshots.jsonl`.
|
||||
|
||||
**Phase 22 item B — cloud rescue (WIRED):** When an event fails and cloud T3 is enabled, `requestCloudRemediation()` feeds the failure trace (SQL filters attempted, row counts, reviewer drift reasons, gap signals, contract terms) to cloud and parses a JSON remediation with new_city / new_state / new_role / new_count / rationale. Event retries once with the pivot. Verified 1/3 rescues succeeded on stress_01 (Gary IN → South Bend IN pivot filled a Welder that local drift-aborted). Sanitizer splits "City, ST" comma-packed outputs so downstream SQL doesn't get `Hammond, IN, IN`.
|
||||
|
||||
### Phase 23: Staffer identity + competence-weighted retrieval (WIRED 2026-04-21)
|
||||
|
||||
Answers "who handled this" as a first-class dimension of the matrix index. Senior staffers' playbooks rank higher than juniors' on similar scenarios via competence × similarity score. Auto-discovers "reliable performer" worker labels via cross-staffer endorsement overlap.
|
||||
|
||||
**Schema (`scenario.ts` ScenarioSpec):**
|
||||
- `contract?: ContractTerms` — deadline, budget_per_hour_max, local_bonus_per_hour, local_bonus_radius_mi, fill_requirement. Propagates into T3 checkpoint + cloud rescue prompts so cloud reasons about trade-offs (pivot-within-radius before budget-pivot-further).
|
||||
- `staffer?: Staffer` — {id, name, tenure_months, role, tool_level}. tool_level controls subsystems available to this run:
|
||||
- `full` — qwen3.5 + qwen3 local + cloud T3 + cloud rescue
|
||||
- `local` — qwen3.5 + qwen3 local + local gpt-oss:20b T3 + rescue
|
||||
- `basic` — **kimi-k2.5 cloud** exec + qwen3 local reviewer + local T3, no rescue
|
||||
- `minimal` — kimi-k2.5 cloud exec + qwen3 local reviewer, NO T3, NO rescue — tests whether playbook inheritance carries knowledge alone
|
||||
|
||||
**KB staffer indexing (`data/_kb/staffers.jsonl`):**
|
||||
- Recomputed per-staffer on every run: total_runs, fill_rate, avg_turns_per_event, avg_citations_per_run, rescue_rate, competence_score.
|
||||
- `competence_score = 0.45·fill_rate + 0.20·turn_efficiency + 0.20·citation_density + 0.15·rescue_rate`. Bounded 0..1.
|
||||
|
||||
**Weighted neighbor retrieval:**
|
||||
- `findNeighbors` in `kb.ts` returns `weighted_score = cosine × max_staffer_competence` (floor 0.3). Senior playbooks rank above junior playbooks on similar scenarios.
|
||||
- `pathway_recommendations` include `best_staffer_id` / `best_staffer_competence` so cloud knows WHOSE playbook it's synthesizing from.
|
||||
|
||||
**Cross-staffer auto-discovery:**
|
||||
- `scripts/kb_staffer_report.py` emits leaderboard + workers endorsed across ≥2 staffers on same signature.
|
||||
- Validated output: Rachel D. Lewis (Welder Nashville) endorsed 12× across 4 staffers; Christina Watson (Machine Op Indianapolis) 11×. These are the highest-confidence "reliable performer" labels the system produced without human tagging.
|
||||
|
||||
**Demo infrastructure:**
|
||||
- `tests/multi-agent/gen_staffer_demo.ts` — 4 personas × 3 contracts = 12 scenario specs.
|
||||
- `scripts/run_staffer_demo.sh` — sequential batch with cloud T3.
|
||||
- `scripts/kb_staffer_report.py` — leaderboard + top/bottom differential + cross-staffer overlap.
|
||||
|
||||
### Phase 24: Observer / Autotune integration (SHIPPED 2026-04-20, commit b95dd86)
|
||||
|
||||
The gap: `lakehouse-observer.service` wrapped MCP :3700, while `tests/multi-agent/scenario.ts` hit gateway :3100 directly. Observer idle at 0 ops across 3600+ cycles — scenarios invisible to ERROR_ANALYZER and PLAYBOOK_BUILDER, autotune running blind to outcomes.
|
||||
|
||||
**What shipped:**
|
||||
- `observer.ts` Bun HTTP listener on `OBSERVER_PORT` (default 3800): `GET /health`, `GET /stats` (totals, by_source, recent scenario digest), `POST /event` for scenario outcomes.
|
||||
- `ObservedOp` carries provenance — `source="scenario" | "mcp"` + `staffer_id` + `sig_hash` + `event_kind` + geo + rescue flags.
|
||||
- `recordExternalOp()` — shared ring-buffer insert; main analyzer + playbook builder no longer care where the op came from.
|
||||
- `persistOp()` fix: old path POSTed to `/ingest/file?name=observed_operations` which has REPLACE semantics (wiped prior ops); now uses append-friendly write-through.
|
||||
|
||||
### Phase 25: Validity windows + playbook retirement (SHIPPED 2026-04-21, commit e0a843d)
|
||||
|
||||
Zep 2026-era finding: temporal validity is the single highest-value memory-hygiene primitive. `PlaybookEntry` gained `schema_fingerprint` / `valid_until` / `retired_at` / `retirement_reason`. `compute_boost_for_filtered_with_role` skips retired + expired before geo/cosine ranking. Two retirement paths: `retire_one(id, reason)` for manual, `retire_on_schema_drift(city, state, fp, reason)` for batch migration sweep. Endpoint: `POST /vectors/playbook_memory/retire`.
|
||||
|
||||
### Phase 26: Mem0 upsert + Letta geo hot cache (SHIPPED 2026-04-21, commit 640db8c)
|
||||
|
||||
Same-day re-seed no longer duplicates rows. `/seed` with `append=true` routes through `upsert_entry` which decides ADD / UPDATE / NOOP on `(operation, day, city, state)`. Playbook_id stays stable on UPDATE so existing citations remain valid. `PlaybookMemory.geo_index: HashMap<(city, state), Vec<idx>>` rebuilt on every mutation; geo-filtered boost queries skip the scan and hit O(1) lookup — sub-ms at current scale, same code path scales to 100K+ entries.
|
||||
|
||||
### Phase 27: Playbook versioning (SHIPPED 2026-04-21)
|
||||
|
||||
`PlaybookEntry` gained `version: u32` (default 1), `parent_id`, `superseded_at`, `superseded_by` — all `#[serde(default)]` so pre-Phase-27 state loads as roots. `revise_entry(parent_id, new_entry)` appends a new version, stamps the parent superseded, rejects revising a retired or already-superseded parent. `history(id)` returns the root→tip chain from any node. Superseded entries excluded from boost (same rule as retired). Endpoints: `POST /vectors/playbook_memory/revise`, `GET /vectors/playbook_memory/history/{id}`. `/status` reports `superseded` as a distinct counter. 8 new tests; 51/51 vectord lib tests green.
|
||||
|
||||
### Phase 28: Agent Gateway + MCP Server (SHIPPED 2026-04-21)
|
||||
|
||||
The agent gateway wraps the Rust substrate with an MCP-first interface (Model Context Protocol), enabling Claude Code, GPT agents, and internal scripts to interact with the lakehouse through named tools rather than raw HTTP. Built on Bun (not Node), serving as the "front door" for all AI consumers.
|
||||
|
||||
**Files:** `mcp-server/index.ts` (2241 lines)
|
||||
|
||||
| Tool | Purpose | Endpoint |
|
||||
|---|---|---|
|
||||
| `search_workers` | Hybrid SQL+vector (the core) | `POST /vectors/hybrid` |
|
||||
| `query_sql` | Analytical SQL on any dataset | `POST /query/sql` |
|
||||
| `match_contract` | Find workers for a job order | `POST /match` |
|
||||
| `get_worker` | Single worker by ID | `POST /worker/:id` |
|
||||
| `rag_question` | Full RAG pipeline | `POST /vectors/rag` |
|
||||
| `log_success` | Record operation → playbook | `POST /log` |
|
||||
| `log_failure` | Record failed fill (negative signal) | `POST /log_failure` |
|
||||
| `get_playbooks` | Retrieve past successes | `GET /playbooks` |
|
||||
| `swap_profile` | Hot-swap model+data context | `POST /profile/:id` |
|
||||
| `vram_status` | GPU introspection | `GET /vram` |
|
||||
|
||||
**HTTP Routes (internal agent consumption):**
|
||||
- `/search` — hybrid search with client blacklist + rate enrichment
|
||||
- `/verify` — claim verification against golden dataset
|
||||
- `/context` — self-orientation for agents
|
||||
- `/clients/:client/blacklist` — per-client worker exclusion
|
||||
- `/memory/query` — unified memory surface (Phase 24 refinement)
|
||||
- `/system/summary` — truthful row counts via SQL
|
||||
- `/models/matrix` — read `config/models.json`
|
||||
|
||||
### Phase 29: Intelligence Suite (SHIPPED 2026-04-21)
|
||||
|
||||
Market intelligence endpoints surfaced through the MCP server — real-world demand signals from public data, cross-referenced with worker bench for staffing gap analysis.
|
||||
|
||||
**Files:** `mcp-server/index.ts` continued
|
||||
|
||||
| Endpoint | Purpose | Data Source |
|
||||
|---|---|---|
|
||||
| `/intelligence/market` | Building permits → demand forecast | Chicago Socrata API |
|
||||
| `/intelligence/staffing_forecast` | 30-day demand by role | Chicago permits + bench |
|
||||
| `/intelligence/permit_contracts` | Permits + Phase 19 ranked candidates | Chicago permits + Workers 500K |
|
||||
| `/intelligence/activity` | Activity feed + learned patterns | successful_playbooks |
|
||||
| `/intelligence/brief` | Parallel analytics across 500K profiles | Workers 500K |
|
||||
| `/intelligence/chat` | Natural language → routed queries | Hybrid + RAG |
|
||||
|
||||
**Market signals captured:**
|
||||
- Largest permits by cost (top 50)
|
||||
- Work type breakout (electrical, mechanical, masonry, plumbing)
|
||||
- Cross-reference with IL bench supply/available/reliable
|
||||
- Demand forecast using $150K/worker industry heuristic
|
||||
|
||||
### Phase 30: Observer + Autotune Integration (SHIPPED 2026-04-20, commit b95dd86)
|
||||
|
||||
The gap: `lakehouse-observer.service` wrapped MCP :3700, while `tests/multi-agent/scenario.ts` hit gateway :3100 directly. Observer idle at 0 ops across 3600+ cycles — scenarios invisible to ERROR_ANALYZER and PLAYBOOK_BUILDER, autotune running blind to outcomes.
|
||||
|
||||
**Files:** `mcp-server/observer.ts` (335 lines)
|
||||
|
||||
**What shipped:**
|
||||
- `observer.ts` Bun HTTP listener on `OBSERVER_PORT` (default 3800): `GET /health`, `GET /stats` (totals, by_source, recent scenario digest), `POST /event` for scenario outcomes.
|
||||
- `ObservedOp` carries provenance — `source="scenario" | "mcp"` + `staffer_id` + `sig_hash` + `event_kind` + geo + rescue flags.
|
||||
- `recordExternalOp()` — shared ring-buffer insert; main analyzer + playbook builder no longer care where the op came from.
|
||||
- `persistOp()` fix: old path POSTed to `/ingest/file?name=observed_operations` which has REPLACE semantics (wiped prior ops); now uses append-friendly write-through to `data/_observer/ops.jsonl`.
|
||||
|
||||
**HTTP API:**
|
||||
- `GET /health` — ring size
|
||||
- `GET /stats` — totals, by_source, recent scenario ops, rescue stats
|
||||
- `POST /event` — record scenario outcomes with provenance
|
||||
|
||||
### Phase 31: Scenario Orchestrator with Tiers (SHIPPED 2026-04-21)
|
||||
|
||||
A day-in-the-life scenario test running six events against the live substrate. Exercises baseline_fill, recurring, expansion, emergency, misplacement. Routes through the same executor/reviewer loop as the single-task orchestrator with event-specific constraints.
|
||||
|
||||
**Files:** `tests/multi-agent/scenario.ts` (1772 lines)
|
||||
|
||||
**Model Matrix (configurable per-tier):**
|
||||
|
||||
| Tier | Purpose | Model | Context Budget |
|
||||
|---|---|---|---|
|
||||
| T1 hot | 50-200 calls/scenario | qwen3.5:latest | 8K, think:false |
|
||||
| T2 review | 5-14 calls/event | qwen3:latest | 8K, think:false |
|
||||
| T3 overview | 1-3 calls/scenario | gpt-oss:120b | 32K, thinking on |
|
||||
| T4 strategic | 1-10 calls/day | gpt-oss:120b | 64K |
|
||||
| T5 gatekeeper | 1-5 calls/day | kimi-k2.6 | 128K, audit-logged |
|
||||
|
||||
**Staffer tool_levels (Phase 23):**
|
||||
- `full` — qwen3.5 + qwen3 local + cloud T3 + cloud rescue
|
||||
- `local` — qwen3.5 + qwen3 local + local gpt-oss:20b T3 + rescue
|
||||
- `basic` — kimi-k2.5 cloud exec + qwen3 local reviewer, no rescue
|
||||
- `minimal` — kimi-k2.5 cloud exec + qwen3 local reviewer, NO T3, NO rescue
|
||||
|
||||
**Key features:**
|
||||
- `generateContinuable()` — handles output overflow with continuation
|
||||
- `generateTreeSplit()` — handles input overflow via sharding
|
||||
- `think:false` flag — disables hidden reasoning for hot-path JSON emitters
|
||||
- Cloud executor routing — `ACTIVE_EXECUTOR_CLOUD` / `ACTIVE_REVIEWER_CLOUD` flags
|
||||
- Cloud rescue (Phase 22B) — `requestCloudRemediation()` on failure
|
||||
- KB integration at start (load recommendation) and end (index + recommend)
|
||||
|
||||
### Phase 32: Knowledge Library + Staffer Indexing (SHIPPED 2026-04-21)
|
||||
|
||||
Meta-layer over Phase 19 playbook_memory. Tracks which configs worked for which playbook signatures.
|
||||
|
||||
**Files:** `tests/multi-agent/kb.ts` (600 lines)
|
||||
|
||||
**Files under `data/_kb/`:**
|
||||
- `signatures.jsonl` — (sig_hash, embedding[], first_seen, last_seen, run_count)
|
||||
- `outcomes.jsonl` — per-run: {sig, run_id, models, ok/total, turns, citations}
|
||||
- `pathway_recommendations.jsonl` — AI-synthesized for next run
|
||||
- `error_corrections.jsonl` — detected fail→succeed pairs
|
||||
- `config_snapshots.jsonl` — history of model changes
|
||||
- `staffers.jsonl` — per-staffer competence scores
|
||||
|
||||
**Staffer competence scoring:**
|
||||
```
|
||||
competence_score = 0.45·fill_rate + 0.20·turn_efficiency + 0.20·citation_density + 0.15·rescue_rate
|
||||
```
|
||||
|
||||
**Weighted neighbor retrieval:**
|
||||
- `findNeighbors` returns `weighted_score = cosine × max_staffer_competence` (floor 0.3)
|
||||
- Senior playbooks rank above junior playbooks on similar scenarios
|
||||
|
||||
### Phase 33: Validity Windows + Playbook Retirement (SHIPPED 2026-04-21)
|
||||
|
||||
Zep 2026-era finding: temporal validity is the single highest-value memory-hygiene primitive.
|
||||
|
||||
**What shipped:**
|
||||
- `PlaybookEntry` gained `schema_fingerprint` / `valid_until` / `retired_at` / `retirement_reason`
|
||||
- `compute_boost_for_filtered_with_role` skips retired + expired before geo/cosine ranking
|
||||
- Two retirement paths: `retire_one(id, reason)` manual, `retire_on_schema_drift(city, state, fp, reason)` batch
|
||||
- Endpoint: `POST /vectors/playbook_memory/retire`
|
||||
|
||||
### Phase 34: Mem0 Upsert + Geo Hot Cache (SHIPPED 2026-04-21, commit 640db8c)
|
||||
|
||||
Same-day re-seed no longer duplicates rows.
|
||||
|
||||
**What shipped:**
|
||||
- `/seed` with `append=true` routes through `upsert_entry` — decides ADD / UPDATE / NOOP on `(operation, day, city, state)`
|
||||
- Playbook_id stays stable on UPDATE so existing citations remain valid
|
||||
- `PlaybookMemory.geo_index: HashMap<(city, state), Vec<idx>>` rebuilt on every mutation
|
||||
- O(1) geo-filtered boost queries — sub-MS at current scale
|
||||
|
||||
### Phase 36: Gateway Seed Concurrency Fix (SHIPPED 2026-04-22)
|
||||
|
||||
Problem: Concurrent `playbook_memory/seed` calls caused socket collisions with the Python AI sidecar. Two staffing coordinators hitting seed simultaneously → one socket error.
|
||||
|
||||
Solution:
|
||||
- Added `embed_semaphore: Arc<Semaphore>` to `VectorState` (permits=1)
|
||||
- Seed endpoint acquires permit before embedding, releases after
|
||||
- Serializes embed calls to sidecar, eliminates collision
|
||||
|
||||
E2E test result: 8.0/10 avg (persist=0 is expected — same day+op = NOOP, not add)
|
||||
|
||||
### Phase 37: Hot-Swap Endpoint Hang (TODO)
|
||||
|
||||
Problem: `POST /vectors/profile/{id}/activate` hangs for 2-4 minutes because it does heavy work synchronously:
|
||||
- Loads embeddings into memory
|
||||
- Builds HNSW indexes
|
||||
- Preloads Ollama model
|
||||
- Auto-provisions buckets
|
||||
|
||||
Current workaround: Client uses 5-second timeout (test passes but doesn't verify hot-swap).
|
||||
|
||||
**Required fix**: Convert to async job pattern:
|
||||
1. Endpoint accepts request, spawns background task, returns job_id immediately
|
||||
2. Client polls `GET /jobs/{job_id}` for status
|
||||
3. Progress events streamed via SSE during activation
|
||||
4. Background task updates job status on completion
|
||||
|
||||
This is a known architectural debt item - not blocking since tests use timeout workaround.
|
||||
|
||||
### Phase 35+: Further horizon
|
||||
|
||||
- PDF OCR for scanned documents (Tesseract integration)
|
||||
- Specialized fine-tuned models per domain (staffing matcher, resume parser)
|
||||
- Video/audio transcript ingest + multimodal embeddings
|
||||
- Neural re-ranker over (query, candidate, outcome) triples — only if Phase 19's statistical feedback plateaus
|
||||
- True distributed query (DataFusion multi-node) — only if single-machine ceilings bite
|
||||
|
||||
---
|
||||
|
||||
@@ -70,3 +70,9 @@ cycle_interval_secs = 120 # periodic wake if no triggers
|
||||
cooldown_between_trials_secs = 10 # min gap between trials
|
||||
min_recall = 0.9 # never promote below this
|
||||
max_trials_per_hour = 20 # hard budget cap
|
||||
|
||||
# Model roster — available for profile hot-swap
|
||||
# qwen3: 8.2B, 40K context, thinking+tools, best for reasoning tasks
|
||||
# qwen2.5: 7B, 8K context, fast, good for SQL generation
|
||||
# mistral: 7B, 8K context, good for general generation
|
||||
# nomic-embed-text: 137M, embedding-only, used by all profiles
|
||||
|
||||
@@ -0,0 +1,866 @@
|
||||
|
||||
=== OVERNIGHT PROOF ===
|
||||
Start: 01:28:59
|
||||
|
||||
STEP 1: Embedding 5K real workers
|
||||
Fetched 5000 rows
|
||||
5000 docs ready
|
||||
Job job-1776407339506: 5000 chunks embedding...
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (0s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...01:30:01 ═══ OVERNIGHT PROOF: step= ═══
|
||||
01:30:01 ═══ OVERNIGHT PROOF: step= ═══
|
||||
unknown: 0/50000 chunks... ... ?: 0/5000 (60s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (120s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
============================================================
|
||||
OVERNIGHT PROOF — steps 2-6 on REAL embeddings
|
||||
============================================================
|
||||
|
||||
STEP 2: HNSW on workers_5k_proof
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... HNSW: 5000 vectors in 5.3s
|
||||
|
||||
STEP 3: Lance
|
||||
Migrate: 5000 rows in 0.0s
|
||||
unknown: 0/50000 chunks... IVF_PQ: built in 2.4s
|
||||
Btree: 0.00s
|
||||
|
||||
STEP 4: Recall on REAL embeddings
|
||||
Eval harness: 30 queries
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... HNSW recall@10: 1.0000 p50: 762us
|
||||
Lance recall@10: 0.9500 p50: 6791us
|
||||
|
||||
STEP 5: 50 autonomous SQL operations
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... 50/50 passed (100%) in 2.9s
|
||||
|
||||
STEP 6: Hybrid search — REAL embeddings + SQL filter
|
||||
method=hybrid_sql_vector sql=6289 results=0
|
||||
|
||||
============================================================
|
||||
OVERNIGHT PROOF — RESULTS
|
||||
============================================================
|
||||
Real embeddings: 5,000 via nomic-embed-text (not random)
|
||||
HNSW recall@10: 1.0000 p50: 762us
|
||||
Lance recall@10: 0.9500 p50: 6791us
|
||||
SQL autonomous: 50/50 (100%)
|
||||
Hybrid search: FAILED
|
||||
============================================================
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (180s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...01:33:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
01:33:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
01:33:01 Overnight proof already complete.
|
||||
01:33:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks... ... ?: 0/5000 (240s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (300s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (360s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...01:36:02 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
01:36:02 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
01:36:02 Overnight proof already complete.
|
||||
01:36:02 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks... ... ?: 0/5000 (420s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (480s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (540s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...01:39:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
01:39:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
01:39:01 Overnight proof already complete.
|
||||
01:39:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks... ... ?: 0/5000 (600s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (660s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (720s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...01:42:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
01:42:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
01:42:01 Overnight proof already complete.
|
||||
01:42:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks... ... ?: 0/5000 (780s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (840s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (900s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...01:45:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
01:45:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
01:45:01 Overnight proof already complete.
|
||||
01:45:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks... ... ?: 0/5000 (960s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (1020s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (1080s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...01:48:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
01:48:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
01:48:01 Overnight proof already complete.
|
||||
01:48:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks... ... ?: 0/5000 (1140s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (1200s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (1260s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...01:51:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
01:51:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
01:51:01 Overnight proof already complete.
|
||||
01:51:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks... ... ?: 0/5000 (1320s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (1380s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... batch 1: TIMEOUT waiting for job job-1776407228716
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (1440s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...01:54:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
01:54:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
01:54:01 Overnight proof already complete.
|
||||
01:54:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks... ... ?: 0/5000 (1500s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (1560s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (1620s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...01:57:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
01:57:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
01:57:01 Overnight proof already complete.
|
||||
01:57:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks... ... ?: 0/5000 (1680s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (1740s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (1800s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:00:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:00:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:00:01 Overnight proof already complete.
|
||||
02:00:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks... ... ?: 0/5000 (1860s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (1920s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (1980s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:03:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:03:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:03:01 Overnight proof already complete.
|
||||
02:03:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks... ... ?: 0/5000 (2040s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (2100s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (2160s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:06:02 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:06:02 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:06:02 Overnight proof already complete.
|
||||
02:06:02 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks... ... ?: 0/5000 (2220s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (2280s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (2340s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:09:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:09:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:09:01 Overnight proof already complete.
|
||||
02:09:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks... ... ?: 0/5000 (2400s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (2460s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (2520s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
|
||||
Batch 2/10: 50000 docs → index 'workers_500k_v2'
|
||||
Job job-1776409866112: 50000 chunks, embedding in background...
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:12:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:12:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:12:01 Overnight proof already complete.
|
||||
02:12:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (2580s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (2640s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (2700s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:15:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:15:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:15:01 Overnight proof already complete.
|
||||
02:15:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (2760s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (2820s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (2880s)
|
||||
unknown: 0/50000 chunks...
|
||||
|
||||
Batch 2/10: 50000 docs → index 'workers_500k_v2'
|
||||
Job job-1776410224649: 50000 chunks, embedding in background...
|
||||
unknown: 0/50000 chunks... batch 2: TIMEOUT waiting for job job-1776408729457
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:18:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:18:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:18:01 Overnight proof already complete.
|
||||
02:18:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... ... ?: 0/5000 (2940s)
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
STEP 2: HNSW on 5K real embeddings
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... HNSW: 5000 vectors in 5.6s
|
||||
|
||||
STEP 3: Lance migrate + IVF_PQ
|
||||
Lance: 5000 rows in 0.0s
|
||||
unknown: 0/50000 chunks... IVF_PQ: built in 3.9s
|
||||
Btree: built in 0.00s
|
||||
|
||||
STEP 4: Recall on REAL embeddings
|
||||
Harness: 30 queries
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... HNSW recall@10: 1.0000 p50: 1327us
|
||||
Lance recall@10: 0.9467 p50: 8154us
|
||||
|
||||
STEP 5: 50 autonomous operations
|
||||
unknown: 0/50000 chunks... SQL ops: 50/50 (100%) in 1.8s
|
||||
|
||||
STEP 6: Hybrid search on real embeddings
|
||||
method=hybrid_sql_vector sql=6289 results=0
|
||||
|
||||
=== OVERNIGHT PROOF COMPLETE: 02:19:20 ===
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:21:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:21:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:21:01 Overnight proof already complete.
|
||||
02:21:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:24:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:24:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:24:01 Overnight proof already complete.
|
||||
02:24:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:27:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:27:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:27:01 Overnight proof already complete.
|
||||
02:27:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:30:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:30:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:30:01 Overnight proof already complete.
|
||||
02:30:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:33:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:33:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:33:01 Overnight proof already complete.
|
||||
02:33:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:36:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:36:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:36:01 Overnight proof already complete.
|
||||
02:36:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:39:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:39:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:39:01 Overnight proof already complete.
|
||||
02:39:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:42:02 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:42:02 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:42:02 Overnight proof already complete.
|
||||
02:42:02 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... batch 3: TIMEOUT waiting for job job-1776410230212
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:45:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:45:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:45:01 Overnight proof already complete.
|
||||
02:45:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:48:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:48:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:48:01 Overnight proof already complete.
|
||||
02:48:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:51:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:51:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:51:01 Overnight proof already complete.
|
||||
02:51:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:54:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:54:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:54:01 Overnight proof already complete.
|
||||
02:54:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...02:57:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:57:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
02:57:01 Overnight proof already complete.
|
||||
02:57:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:00:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:00:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:00:01 Overnight proof already complete.
|
||||
03:00:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
|
||||
Batch 3/10: 50000 docs → index 'workers_500k_v3'
|
||||
Job job-1776412867733: 50000 chunks, embedding in background...
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:03:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:03:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:03:01 Overnight proof already complete.
|
||||
03:03:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:06:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:06:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:06:01 Overnight proof already complete.
|
||||
03:06:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
|
||||
Batch 3/10: 50000 docs → index 'workers_500k_v3'
|
||||
Job job-1776413226242: 50000 chunks, embedding in background...
|
||||
unknown: 0/50000 chunks... batch 4: TIMEOUT waiting for job job-1776411730966
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:09:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:09:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:09:01 Overnight proof already complete.
|
||||
03:09:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:12:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:12:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:12:01 Overnight proof already complete.
|
||||
03:12:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:15:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:15:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:15:01 Overnight proof already complete.
|
||||
03:15:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:18:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:18:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:18:01 Overnight proof already complete.
|
||||
03:18:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:21:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:21:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:21:01 Overnight proof already complete.
|
||||
03:21:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:24:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:24:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:24:01 Overnight proof already complete.
|
||||
03:24:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:27:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:27:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:27:01 Overnight proof already complete.
|
||||
03:27:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:30:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:30:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:30:01 Overnight proof already complete.
|
||||
03:30:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... batch 5: TIMEOUT waiting for job job-1776413231714
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:33:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:33:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:33:01 Overnight proof already complete.
|
||||
03:33:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:36:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:36:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:36:01 Overnight proof already complete.
|
||||
03:36:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:39:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:39:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:39:01 Overnight proof already complete.
|
||||
03:39:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:42:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:42:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:42:01 Overnight proof already complete.
|
||||
03:42:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:45:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:45:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:45:01 Overnight proof already complete.
|
||||
03:45:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:48:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:48:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:48:01 Overnight proof already complete.
|
||||
03:48:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:51:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:51:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:51:01 Overnight proof already complete.
|
||||
03:51:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
|
||||
Batch 4/10: 50000 docs → index 'workers_500k_v4'
|
||||
Job job-1776415869343: 50000 chunks, embedding in background...
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:54:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:54:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:54:01 Overnight proof already complete.
|
||||
03:54:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...03:57:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:57:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
03:57:01 Overnight proof already complete.
|
||||
03:57:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
|
||||
Batch 4/10: 50000 docs → index 'workers_500k_v4'
|
||||
Job job-1776416227836: 50000 chunks, embedding in background...
|
||||
unknown: 0/50000 chunks... batch 6: TIMEOUT waiting for job job-1776414732507
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:00:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:00:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:00:01 Overnight proof already complete.
|
||||
04:00:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:03:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:03:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:03:01 Overnight proof already complete.
|
||||
04:03:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:06:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:06:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:06:01 Overnight proof already complete.
|
||||
04:06:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:09:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:09:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:09:01 Overnight proof already complete.
|
||||
04:09:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:12:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:12:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:12:01 Overnight proof already complete.
|
||||
04:12:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:15:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:15:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:15:01 Overnight proof already complete.
|
||||
04:15:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:18:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:18:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:18:01 Overnight proof already complete.
|
||||
04:18:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:21:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:21:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:21:01 Overnight proof already complete.
|
||||
04:21:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... batch 7: TIMEOUT waiting for job job-1776416233259
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:24:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:24:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:24:01 Overnight proof already complete.
|
||||
04:24:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:27:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:27:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:27:01 Overnight proof already complete.
|
||||
04:27:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:30:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:30:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:30:01 Overnight proof already complete.
|
||||
04:30:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:33:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:33:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:33:01 Overnight proof already complete.
|
||||
04:33:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:36:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:36:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:36:01 Overnight proof already complete.
|
||||
04:36:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:39:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:39:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:39:01 Overnight proof already complete.
|
||||
04:39:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
|
||||
Batch 5/10: 50000 docs → index 'workers_500k_v5'
|
||||
Job job-1776418870939: 50000 chunks, embedding in background...
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:42:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:42:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:42:01 Overnight proof already complete.
|
||||
04:42:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:45:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:45:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:45:01 Overnight proof already complete.
|
||||
04:45:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
|
||||
Batch 5/10: 50000 docs → index 'workers_500k_v5'
|
||||
Job job-1776419229418: 50000 chunks, embedding in background...
|
||||
unknown: 0/50000 chunks... batch 8: TIMEOUT waiting for job job-1776417734019
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:48:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:48:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:48:01 Overnight proof already complete.
|
||||
04:48:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:51:02 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:51:02 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:51:02 Overnight proof already complete.
|
||||
04:51:02 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:54:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:54:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:54:01 Overnight proof already complete.
|
||||
04:54:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...04:57:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:57:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
04:57:01 Overnight proof already complete.
|
||||
04:57:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:00:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:00:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:00:01 Overnight proof already complete.
|
||||
05:00:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:03:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:03:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:03:01 Overnight proof already complete.
|
||||
05:03:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:06:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:06:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:06:01 Overnight proof already complete.
|
||||
05:06:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:09:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:09:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:09:01 Overnight proof already complete.
|
||||
05:09:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:12:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:12:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:12:01 Overnight proof already complete.
|
||||
05:12:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... batch 9: TIMEOUT waiting for job job-1776419234766
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:15:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:15:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:15:01 Overnight proof already complete.
|
||||
05:15:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:18:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:18:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:18:01 Overnight proof already complete.
|
||||
05:18:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:21:02 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:21:02 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:21:02 Overnight proof already complete.
|
||||
05:21:02 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:24:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:24:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:24:01 Overnight proof already complete.
|
||||
05:24:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:27:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:27:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:27:01 Overnight proof already complete.
|
||||
05:27:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:30:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:30:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:30:01 Overnight proof already complete.
|
||||
05:30:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
|
||||
Batch 6/10: 50000 docs → index 'workers_500k_v6'
|
||||
Job job-1776421872553: 50000 chunks, embedding in background...
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:33:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:33:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:33:01 Overnight proof already complete.
|
||||
05:33:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:36:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:36:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:36:01 Overnight proof already complete.
|
||||
05:36:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
|
||||
Batch 6/10: 50000 docs → index 'workers_500k_v6'
|
||||
Job job-1776422230966: 50000 chunks, embedding in background...
|
||||
unknown: 0/50000 chunks... batch 10: TIMEOUT waiting for job job-1776420735517
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:39:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:39:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:39:01 Overnight proof already complete.
|
||||
05:39:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:42:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:42:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:42:01 Overnight proof already complete.
|
||||
05:42:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:45:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:45:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:45:01 Overnight proof already complete.
|
||||
05:45:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:48:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:48:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:48:01 Overnight proof already complete.
|
||||
05:48:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:51:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:51:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:51:01 Overnight proof already complete.
|
||||
05:51:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:54:02 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:54:02 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:54:02 Overnight proof already complete.
|
||||
05:54:02 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...05:57:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:57:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
05:57:01 Overnight proof already complete.
|
||||
05:57:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:00:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:00:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:00:01 Overnight proof already complete.
|
||||
06:00:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... batch 11: TIMEOUT waiting for job job-1776422236270
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:03:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:03:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:03:01 Overnight proof already complete.
|
||||
06:03:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:06:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:06:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:06:01 Overnight proof already complete.
|
||||
06:06:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:09:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:09:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:09:01 Overnight proof already complete.
|
||||
06:09:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:12:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:12:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:12:01 Overnight proof already complete.
|
||||
06:12:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:15:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:15:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:15:01 Overnight proof already complete.
|
||||
06:15:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:18:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:18:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:18:01 Overnight proof already complete.
|
||||
06:18:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:21:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:21:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:21:01 Overnight proof already complete.
|
||||
06:21:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
|
||||
Batch 7/10: 50000 docs → index 'workers_500k_v7'
|
||||
Job job-1776424874127: 50000 chunks, embedding in background...
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:24:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:24:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:24:01 Overnight proof already complete.
|
||||
06:24:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:27:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:27:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:27:01 Overnight proof already complete.
|
||||
06:27:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
|
||||
Batch 7/10: 50000 docs → index 'workers_500k_v7'
|
||||
Job job-1776425232575: 50000 chunks, embedding in background...
|
||||
unknown: 0/50000 chunks... batch 12: TIMEOUT waiting for job job-1776423737035
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:30:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:30:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:30:01 Overnight proof already complete.
|
||||
06:30:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:33:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:33:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:33:01 Overnight proof already complete.
|
||||
06:33:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:36:02 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:36:02 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:36:02 Overnight proof already complete.
|
||||
06:36:02 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:39:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:39:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:39:01 Overnight proof already complete.
|
||||
06:39:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:42:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:42:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:42:01 Overnight proof already complete.
|
||||
06:42:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:45:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:45:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:45:01 Overnight proof already complete.
|
||||
06:45:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:48:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:48:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:48:01 Overnight proof already complete.
|
||||
06:48:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:51:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:51:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:51:01 Overnight proof already complete.
|
||||
06:51:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... batch 13: TIMEOUT waiting for job job-1776425237798
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:54:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:54:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:54:01 Overnight proof already complete.
|
||||
06:54:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...06:57:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:57:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
06:57:01 Overnight proof already complete.
|
||||
06:57:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:00:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:00:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:00:01 Overnight proof already complete.
|
||||
07:00:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:03:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:03:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:03:01 Overnight proof already complete.
|
||||
07:03:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:06:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:06:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:06:01 Overnight proof already complete.
|
||||
07:06:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:09:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:09:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:09:01 Overnight proof already complete.
|
||||
07:09:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
|
||||
Batch 8/10: 50000 docs → index 'workers_500k_v8'
|
||||
Job job-1776427875702: 50000 chunks, embedding in background...
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:12:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:12:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:12:01 Overnight proof already complete.
|
||||
07:12:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:15:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:15:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:15:01 Overnight proof already complete.
|
||||
07:15:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
|
||||
Batch 8/10: 50000 docs → index 'workers_500k_v8'
|
||||
Job job-1776428234147: 50000 chunks, embedding in background...
|
||||
unknown: 0/50000 chunks... batch 14: TIMEOUT waiting for job job-1776426738583
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:18:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:18:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:18:01 Overnight proof already complete.
|
||||
07:18:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:21:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:21:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:21:01 Overnight proof already complete.
|
||||
07:21:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:24:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:24:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:24:01 Overnight proof already complete.
|
||||
07:24:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:27:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:27:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:27:01 Overnight proof already complete.
|
||||
07:27:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:30:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:30:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:30:01 Overnight proof already complete.
|
||||
07:30:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:33:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:33:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:33:01 Overnight proof already complete.
|
||||
07:33:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:36:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:36:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:36:01 Overnight proof already complete.
|
||||
07:36:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:39:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:39:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:39:01 Overnight proof already complete.
|
||||
07:39:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:42:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:42:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:42:01 Overnight proof already complete.
|
||||
07:42:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... batch 15: TIMEOUT waiting for job job-1776428239296
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:45:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:45:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:45:01 Overnight proof already complete.
|
||||
07:45:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:48:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:48:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:48:01 Overnight proof already complete.
|
||||
07:48:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:51:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:51:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:51:01 Overnight proof already complete.
|
||||
07:51:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:54:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:54:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:54:01 Overnight proof already complete.
|
||||
07:54:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...07:57:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:57:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
07:57:01 Overnight proof already complete.
|
||||
07:57:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:00:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:00:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:00:01 Overnight proof already complete.
|
||||
08:00:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
|
||||
Batch 9/10: 50000 docs → index 'workers_500k_v9'
|
||||
Job job-1776430877319: 50000 chunks, embedding in background...
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:03:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:03:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:03:01 Overnight proof already complete.
|
||||
08:03:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:06:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:06:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:06:01 Overnight proof already complete.
|
||||
08:06:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
|
||||
Batch 9/10: 50000 docs → index 'workers_500k_v9'
|
||||
Job job-1776431235754: 50000 chunks, embedding in background...
|
||||
unknown: 0/50000 chunks... batch 16: TIMEOUT waiting for job job-1776429740065
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:09:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:09:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:09:01 Overnight proof already complete.
|
||||
08:09:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:12:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:12:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:12:01 Overnight proof already complete.
|
||||
08:12:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:15:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:15:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:15:01 Overnight proof already complete.
|
||||
08:15:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:18:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:18:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:18:01 Overnight proof already complete.
|
||||
08:18:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:21:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:21:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:21:01 Overnight proof already complete.
|
||||
08:21:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:24:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:24:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:24:01 Overnight proof already complete.
|
||||
08:24:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:27:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:27:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:27:01 Overnight proof already complete.
|
||||
08:27:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:30:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:30:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:30:01 Overnight proof already complete.
|
||||
08:30:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... batch 17: TIMEOUT waiting for job job-1776431240798
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:33:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:33:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:33:01 Overnight proof already complete.
|
||||
08:33:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:36:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:36:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:36:01 Overnight proof already complete.
|
||||
08:36:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:39:02 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:39:02 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:39:02 Overnight proof already complete.
|
||||
08:39:02 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:42:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:42:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:42:01 Overnight proof already complete.
|
||||
08:42:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:45:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:45:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:45:01 Overnight proof already complete.
|
||||
08:45:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:48:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:48:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:48:01 Overnight proof already complete.
|
||||
08:48:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:51:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:51:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:51:01 Overnight proof already complete.
|
||||
08:51:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
|
||||
Batch 10/10: 50000 docs → index 'workers_500k_v10'
|
||||
Job job-1776433878963: 50000 chunks, embedding in background...
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:54:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:54:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:54:01 Overnight proof already complete.
|
||||
08:54:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...08:57:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:57:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
08:57:01 Overnight proof already complete.
|
||||
08:57:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... batch 18: TIMEOUT waiting for job job-1776432741532
|
||||
|
||||
|
||||
Batch 10/10: 50000 docs → index 'workers_500k_v10'
|
||||
Job job-1776434237339: 50000 chunks, embedding in background...
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...09:00:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:00:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:00:01 Overnight proof already complete.
|
||||
09:00:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...09:03:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:03:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:03:01 Overnight proof already complete.
|
||||
09:03:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...09:06:02 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:06:02 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:06:02 Overnight proof already complete.
|
||||
09:06:02 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...09:09:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:09:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:09:01 Overnight proof already complete.
|
||||
09:09:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...09:12:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:12:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:12:01 Overnight proof already complete.
|
||||
09:12:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...09:15:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:15:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:15:01 Overnight proof already complete.
|
||||
09:15:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...09:18:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:18:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:18:01 Overnight proof already complete.
|
||||
09:18:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...09:21:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:21:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:21:01 Overnight proof already complete.
|
||||
09:21:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks... batch 19: TIMEOUT waiting for job job-1776434242270
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...09:24:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:24:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:24:01 Overnight proof already complete.
|
||||
09:24:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...09:27:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:27:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:27:01 Overnight proof already complete.
|
||||
09:27:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...09:30:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:30:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:30:01 Overnight proof already complete.
|
||||
09:30:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...09:33:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:33:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:33:01 Overnight proof already complete.
|
||||
09:33:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...09:36:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:36:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:36:01 Overnight proof already complete.
|
||||
09:36:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...09:39:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:39:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:39:01 Overnight proof already complete.
|
||||
09:39:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
|
||||
All batches submitted. Checking indexes...
|
||||
STEP 1 COMPLETE
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...09:42:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:42:01 ═══ OVERNIGHT PROOF: step=done ═══
|
||||
09:42:01 Overnight proof already complete.
|
||||
09:42:01 Overnight proof already complete.
|
||||
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
unknown: 0/50000 chunks...
|
||||
@@ -0,0 +1,150 @@
|
||||
2026-04-17 01:06:09 ═══ Scale test heartbeat: step= ═══
|
||||
2026-04-17 01:06:09 Unknown state: . Resetting to start.
|
||||
2026-04-17 01:06:09 Heartbeat done. Next step: start
|
||||
2026-04-17 01:06:21 ═══ Scale test heartbeat: step=start ═══
|
||||
2026-04-17 01:06:21 Step 1: Registering 10M vector index in catalog...
|
||||
2026-04-17 01:06:21 Parquet exists: 29G
|
||||
2026-04-17 01:06:21 Heartbeat done. Next step: migrate_lance
|
||||
2026-04-17 01:08:01 ═══ Scale test heartbeat: step=migrate_lance ═══
|
||||
═══ Scale test heartbeat: step=migrate_lance ═══
|
||||
2026-04-17 01:08:01 Step 2: Migrating 10M vectors Parquet → Lance...
|
||||
Step 2: Migrating 10M vectors Parquet → Lance...
|
||||
2026-04-17 01:08:01 This will take several minutes for 28.8 GB...
|
||||
This will take several minutes for 28.8 GB...
|
||||
2026-04-17 01:08:01 Migration via API needs index registered. Using direct Lance path...
|
||||
Migration via API needs index registered. Using direct Lance path...
|
||||
Lance migration needs to read 28.8GB Parquet — this takes time...
|
||||
Starting migration...
|
||||
Error: HTTP Error 404: Not Found
|
||||
Attempting direct Lance write...
|
||||
2026-04-17 01:08:01 Heartbeat done. Next step: check_lance
|
||||
Heartbeat done. Next step: check_lance
|
||||
error: externally-managed-environment
|
||||
|
||||
× This environment is externally managed
|
||||
╰─> To install Python packages system-wide, try apt install
|
||||
python3-xyz, where xyz is the package you are trying to
|
||||
install.
|
||||
|
||||
If you wish to install a non-Debian-packaged Python package,
|
||||
create a virtual environment using python3 -m venv path/to/venv.
|
||||
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
|
||||
sure you have python3-full installed.
|
||||
|
||||
If you wish to install a non-Debian packaged Python application,
|
||||
it may be easiest to use pipx install xyz, which will manage a
|
||||
virtual environment for you. Make sure you have pipx installed.
|
||||
|
||||
See /usr/share/doc/python3.13/README.venv for more information.
|
||||
|
||||
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
|
||||
hint: See PEP 668 for the detailed specification.
|
||||
Traceback (most recent call last):
|
||||
File "<string>", line 11, in <module>
|
||||
import lance
|
||||
ModuleNotFoundError: No module named 'lance'
|
||||
|
||||
During handling of the above exception, another exception occurred:
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "<string>", line 19, in <module>
|
||||
import lance
|
||||
ModuleNotFoundError: No module named 'lance'
|
||||
Missing dep: No module named 'lance'
|
||||
Installing lance...
|
||||
2026-04-17 01:10:01 ═══ Scale test heartbeat: step=check_lance ═══
|
||||
═══ Scale test heartbeat: step=check_lance ═══
|
||||
2026-04-17 01:10:01 Step 2b: Checking Lance dataset status...
|
||||
Step 2b: Checking Lance dataset status...
|
||||
2026-04-17 01:10:02 Lance dataset not ready yet. Will retry on next heartbeat.
|
||||
Lance dataset not ready yet. Will retry on next heartbeat.
|
||||
2026-04-17 01:10:02 Heartbeat done. Next step: check_lance
|
||||
Heartbeat done. Next step: check_lance
|
||||
2026-04-17 01:12:01 ═══ Scale test heartbeat: step=check_lance ═══
|
||||
═══ Scale test heartbeat: step=check_lance ═══
|
||||
2026-04-17 01:12:01 Step 2b: Checking Lance dataset status...
|
||||
Step 2b: Checking Lance dataset status...
|
||||
2026-04-17 01:12:01 Lance dataset: 8000000 rows
|
||||
Lance dataset: 8000000 rows
|
||||
2026-04-17 01:12:01 Heartbeat done. Next step: build_index
|
||||
Heartbeat done. Next step: build_index
|
||||
Migrating 10,000,000 vectors to Lance...
|
||||
500,000 / 10,000,000 (117,052/sec ETA 81s)
|
||||
1,000,000 / 10,000,000 (121,674/sec ETA 74s)
|
||||
1,500,000 / 10,000,000 (123,846/sec ETA 69s)
|
||||
2,000,000 / 10,000,000 (124,296/sec ETA 64s)
|
||||
2,500,000 / 10,000,000 (124,056/sec ETA 60s)
|
||||
3,000,000 / 10,000,000 (124,131/sec ETA 56s)
|
||||
3,500,000 / 10,000,000 (124,769/sec ETA 52s)
|
||||
4,000,000 / 10,000,000 (125,028/sec ETA 48s)
|
||||
4,500,000 / 10,000,000 (125,375/sec ETA 44s)
|
||||
5,000,000 / 10,000,000 (125,476/sec ETA 40s)
|
||||
5,500,000 / 10,000,000 (125,140/sec ETA 36s)
|
||||
6,000,000 / 10,000,000 (124,899/sec ETA 32s)
|
||||
6,500,000 / 10,000,000 (124,355/sec ETA 28s)
|
||||
7,000,000 / 10,000,000 (123,762/sec ETA 24s)
|
||||
7,500,000 / 10,000,000 (123,050/sec ETA 20s)
|
||||
8,000,000 / 10,000,000 (122,744/sec ETA 16s)
|
||||
8,500,000 / 10,000,000 (122,164/sec ETA 12s)
|
||||
9,000,000 / 10,000,000 (121,839/sec ETA 8s)
|
||||
9,500,000 / 10,000,000 (121,655/sec ETA 4s)
|
||||
10,000,000 / 10,000,000 (121,529/sec ETA 0s)
|
||||
Done: 10,000,000 rows in 82s
|
||||
Verified: 10,000,000 rows in Lance
|
||||
2026-04-17 01:14:01 ═══ Scale test heartbeat: step=build_index ═══
|
||||
═══ Scale test heartbeat: step=build_index ═══
|
||||
2026-04-17 01:14:01 Step 3: Building IVF_PQ index on 10M Lance dataset...
|
||||
Step 3: Building IVF_PQ index on 10M Lance dataset...
|
||||
2026-04-17 01:14:01 Using tuned config: 3162 partitions (√10M), 8 bits, 192 sub_vectors
|
||||
Using tuned config: 3162 partitions (√10M), 8 bits, 192 sub_vectors
|
||||
Fri Apr 17 01:16:01 AM CDT 2026 Already running (pid 957071)
|
||||
[2026-04-17T06:16:02Z WARN lance::index::vector::builder] partition 2174 is empty, skipping
|
||||
Dataset: 10,000,000 rows
|
||||
Building IVF_PQ: 3162 partitions, 8 bits, 192 sub_vectors...
|
||||
Index built in 173s
|
||||
|
||||
=== Search benchmark: 10 queries on 10M vectors ===
|
||||
First query: 19ms, 10 hits
|
||||
Top hit: VEC-2662261
|
||||
p50=5ms p95=19ms avg=6ms
|
||||
All 10 searches completed on 10M vectors
|
||||
|
||||
═══════════════════════════════════════════════════════════
|
||||
10M VECTOR SCALE TEST — RESULTS
|
||||
═══════════════════════════════════════════════════════════
|
||||
Vectors: 10,000,000
|
||||
Dimensions: 768
|
||||
Storage: 30 GB (Lance on disk)
|
||||
IVF_PQ build: 173 seconds (3162 partitions, 192 sub_vectors)
|
||||
Search p50: 5ms
|
||||
Search p95: 19ms
|
||||
|
||||
HNSW at 10M would need: 29 GB RAM (past ceiling)
|
||||
Lance at 10M: 30 GB disk, 5ms search
|
||||
|
||||
THIS IS THE PROOF: Lance handles what HNSW can't.
|
||||
═══════════════════════════════════════════════════════════
|
||||
|
||||
Fri Apr 17 01:18:01 AM CDT 2026 Already running (pid 957071)
|
||||
2026-04-17 01:19:10 IVF_PQ built in 309.85242s
|
||||
IVF_PQ built in 309.85242s
|
||||
2026-04-17 01:19:10 Heartbeat done. Next step: search_test
|
||||
Heartbeat done. Next step: search_test
|
||||
2026-04-17 01:20:01 ═══ Scale test heartbeat: step=search_test ═══
|
||||
═══ Scale test heartbeat: step=search_test ═══
|
||||
2026-04-17 01:20:01 Step 4: Search benchmark on 10M vectors...
|
||||
Step 4: Search benchmark on 10M vectors...
|
||||
Running 10 searches on 10M Lance dataset...
|
||||
Search 0 failed: HTTP Error 422: Unprocessable Entity
|
||||
Search 1 failed: HTTP Error 422: Unprocessable Entity
|
||||
Search 2 failed: HTTP Error 422: Unprocessable Entity
|
||||
Search 3 failed: HTTP Error 422: Unprocessable Entity
|
||||
Search 4 failed: HTTP Error 422: Unprocessable Entity
|
||||
Search 5 failed: HTTP Error 422: Unprocessable Entity
|
||||
Search 6 failed: HTTP Error 422: Unprocessable Entity
|
||||
Search 7 failed: HTTP Error 422: Unprocessable Entity
|
||||
Search 8 failed: HTTP Error 422: Unprocessable Entity
|
||||
Search 9 failed: HTTP Error 422: Unprocessable Entity
|
||||
No successful searches
|
||||
2026-04-17 01:20:01 Heartbeat done. Next step: hot_swap_test
|
||||
Heartbeat done. Next step: hot_swap_test
|
||||
@@ -0,0 +1,34 @@
|
||||
# dependencies (bun install)
|
||||
node_modules
|
||||
|
||||
# output
|
||||
out
|
||||
dist
|
||||
*.tgz
|
||||
|
||||
# code coverage
|
||||
coverage
|
||||
*.lcov
|
||||
|
||||
# logs
|
||||
logs
|
||||
_.log
|
||||
report.[0-9]_.[0-9]_.[0-9]_.[0-9]_.json
|
||||
|
||||
# dotenv environment variable files
|
||||
.env
|
||||
.env.development.local
|
||||
.env.test.local
|
||||
.env.production.local
|
||||
.env.local
|
||||
|
||||
# caches
|
||||
.eslintcache
|
||||
.cache
|
||||
*.tsbuildinfo
|
||||
|
||||
# IntelliJ based IDEs
|
||||
.idea
|
||||
|
||||
# Finder (MacOS) folder config
|
||||
.DS_Store
|
||||
@@ -0,0 +1,110 @@
|
||||
# Lakehouse Agent Instructions
|
||||
|
||||
You are connected to a staffing intelligence system. Your job is to answer staffing questions, match workers to contracts, and track what works.
|
||||
|
||||
## Gateway
|
||||
|
||||
All tools are at `http://localhost:3700`. POST JSON, get JSON back.
|
||||
|
||||
## Tools
|
||||
|
||||
### `/search` — Hybrid SQL+Vector Search (use this most)
|
||||
Find workers matching structured criteria + semantic meaning.
|
||||
```json
|
||||
POST /search
|
||||
{
|
||||
"question": "reliable forklift operators with hazmat certification",
|
||||
"sql_filter": "role = 'Forklift Operator' AND state = 'IL' AND reliability > 0.8",
|
||||
"top_k": 5
|
||||
}
|
||||
```
|
||||
Every result is SQL-verified against the database. Trust the `sources` array — those workers exist with those exact skills and scores.
|
||||
|
||||
### `/sql` — Direct SQL Query
|
||||
For exact counts, aggregations, and structured lookups.
|
||||
```json
|
||||
POST /sql
|
||||
{ "sql": "SELECT role, COUNT(*) cnt FROM ethereal_workers GROUP BY role ORDER BY cnt DESC LIMIT 10" }
|
||||
```
|
||||
Tables: `ethereal_workers` (10K), `workers_100k` (100K), `candidates` (100K), `timesheets` (1M), `placements` (50K), `call_log` (800K), `email_log` (500K), `job_orders` (15K), `clients` (2K).
|
||||
|
||||
### `/match` — Match Workers to a Contract
|
||||
```json
|
||||
POST /match
|
||||
{
|
||||
"role": "Machine Operator",
|
||||
"state": "IN",
|
||||
"min_reliability": 0.8,
|
||||
"required_certs": ["OSHA-10"],
|
||||
"headcount": 5
|
||||
}
|
||||
```
|
||||
Returns qualified, SQL-verified workers ranked by semantic fit.
|
||||
|
||||
### `/worker/:id` — Get One Worker
|
||||
```
|
||||
GET /worker/4925
|
||||
```
|
||||
Returns all fields: name, role, city, skills, certifications, scores, communications.
|
||||
|
||||
### `/ask` — RAG Question
|
||||
For open-ended questions. Embeds your question, searches vector index, generates answer.
|
||||
```json
|
||||
POST /ask
|
||||
{ "question": "What kinds of workers do we have in Ohio?" }
|
||||
```
|
||||
|
||||
### `/log` — Record What Worked
|
||||
After a successful operation, log it. Future runs can query past successes.
|
||||
```json
|
||||
POST /log
|
||||
{
|
||||
"operation": "Filled 3 forklift positions in Chicago",
|
||||
"approach": "hybrid search: sql_filter role+state+reliability, vector rank by skills",
|
||||
"result": "3/3 filled, all verified, client satisfied"
|
||||
}
|
||||
```
|
||||
|
||||
### `/playbooks` — Learn From Past Success
|
||||
Before starting a task, check what worked before.
|
||||
```
|
||||
GET /playbooks?keyword=forklift
|
||||
```
|
||||
|
||||
### `/profile/:id` — Swap Model Profile
|
||||
Switch which Ollama model + data context is active.
|
||||
```
|
||||
POST /profile/agent-parquet (HNSW backend, qwen2.5)
|
||||
POST /profile/agent-lance (Lance IVF_PQ backend, mistral)
|
||||
```
|
||||
|
||||
### `/vram` — GPU Status
|
||||
```
|
||||
GET /vram
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
1. **Never hallucinate.** Only state facts that appear in tool responses. If the data doesn't support an answer, say so.
|
||||
2. **SQL first for structured questions.** "How many X in Y?" → use `/sql`. Don't guess counts.
|
||||
3. **Hybrid for matching.** When finding workers for a contract, use `/search` or `/match` with `sql_filter` so results are verified.
|
||||
4. **Log success.** After completing a task successfully, call `/log` so future agents can learn from it.
|
||||
5. **Check playbooks.** Before a complex task, call `/playbooks` to see if a similar task has been done before.
|
||||
6. **Verify before communicating.** Before drafting a message to a worker, confirm their details via `/worker/:id`.
|
||||
|
||||
## Workflow for Contract Filling
|
||||
|
||||
1. `GET /playbooks?keyword={role}` — check if this type was filled before
|
||||
2. `POST /match` with role, state, min_reliability, required_certs
|
||||
3. For each match: `GET /worker/:id` to confirm details
|
||||
4. Draft communication using confirmed worker details
|
||||
5. `POST /log` with outcome
|
||||
|
||||
## Available Profiles
|
||||
|
||||
| Profile | Backend | Model | Best for |
|
||||
|---|---|---|---|
|
||||
| `agent-parquet` | HNSW (RAM) | qwen2.5 | Fast precise search, <100K vectors |
|
||||
| `agent-lance` | IVF_PQ (disk) | mistral | Large scale, append-heavy, random access |
|
||||
|
||||
Swap when you need different capabilities. Check `/vram` before swapping.
|
||||
@@ -0,0 +1,106 @@
|
||||
|
||||
Default to using Bun instead of Node.js.
|
||||
|
||||
- Use `bun <file>` instead of `node <file>` or `ts-node <file>`
|
||||
- Use `bun test` instead of `jest` or `vitest`
|
||||
- Use `bun build <file.html|file.ts|file.css>` instead of `webpack` or `esbuild`
|
||||
- Use `bun install` instead of `npm install` or `yarn install` or `pnpm install`
|
||||
- Use `bun run <script>` instead of `npm run <script>` or `yarn run <script>` or `pnpm run <script>`
|
||||
- Use `bunx <package> <command>` instead of `npx <package> <command>`
|
||||
- Bun automatically loads .env, so don't use dotenv.
|
||||
|
||||
## APIs
|
||||
|
||||
- `Bun.serve()` supports WebSockets, HTTPS, and routes. Don't use `express`.
|
||||
- `bun:sqlite` for SQLite. Don't use `better-sqlite3`.
|
||||
- `Bun.redis` for Redis. Don't use `ioredis`.
|
||||
- `Bun.sql` for Postgres. Don't use `pg` or `postgres.js`.
|
||||
- `WebSocket` is built-in. Don't use `ws`.
|
||||
- Prefer `Bun.file` over `node:fs`'s readFile/writeFile
|
||||
- Bun.$`ls` instead of execa.
|
||||
|
||||
## Testing
|
||||
|
||||
Use `bun test` to run tests.
|
||||
|
||||
```ts#index.test.ts
|
||||
import { test, expect } from "bun:test";
|
||||
|
||||
test("hello world", () => {
|
||||
expect(1).toBe(1);
|
||||
});
|
||||
```
|
||||
|
||||
## Frontend
|
||||
|
||||
Use HTML imports with `Bun.serve()`. Don't use `vite`. HTML imports fully support React, CSS, Tailwind.
|
||||
|
||||
Server:
|
||||
|
||||
```ts#index.ts
|
||||
import index from "./index.html"
|
||||
|
||||
Bun.serve({
|
||||
routes: {
|
||||
"/": index,
|
||||
"/api/users/:id": {
|
||||
GET: (req) => {
|
||||
return new Response(JSON.stringify({ id: req.params.id }));
|
||||
},
|
||||
},
|
||||
},
|
||||
// optional websocket support
|
||||
websocket: {
|
||||
open: (ws) => {
|
||||
ws.send("Hello, world!");
|
||||
},
|
||||
message: (ws, message) => {
|
||||
ws.send(message);
|
||||
},
|
||||
close: (ws) => {
|
||||
// handle close
|
||||
}
|
||||
},
|
||||
development: {
|
||||
hmr: true,
|
||||
console: true,
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
HTML files can import .tsx, .jsx or .js files directly and Bun's bundler will transpile & bundle automatically. `<link>` tags can point to stylesheets and Bun's CSS bundler will bundle.
|
||||
|
||||
```html#index.html
|
||||
<html>
|
||||
<body>
|
||||
<h1>Hello, world!</h1>
|
||||
<script type="module" src="./frontend.tsx"></script>
|
||||
</body>
|
||||
</html>
|
||||
```
|
||||
|
||||
With the following `frontend.tsx`:
|
||||
|
||||
```tsx#frontend.tsx
|
||||
import React from "react";
|
||||
import { createRoot } from "react-dom/client";
|
||||
|
||||
// import .css files directly and it works
|
||||
import './index.css';
|
||||
|
||||
const root = createRoot(document.body);
|
||||
|
||||
export default function Frontend() {
|
||||
return <h1>Hello, world!</h1>;
|
||||
}
|
||||
|
||||
root.render(<Frontend />);
|
||||
```
|
||||
|
||||
Then, run index.ts
|
||||
|
||||
```sh
|
||||
bun --hot ./index.ts
|
||||
```
|
||||
|
||||
For more information, read the Bun API docs in `node_modules/bun-types/docs/**.mdx`.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user