100K embedding: supervisor achieves 67.6/sec (57% faster than single pipeline)

- 4 parallel pipelines on i9 + A4000 via Ollama
- Previous single-pipeline: 43/sec, 39min for 100K
- Supervisor: 67.6/sec, 22min for 100K
- Previous 100K attempt failed at 97K (no retry) — supervisor handles this
- Checkpointing every 1000 chunks for crash recovery
- Round-robin retry on batch failure (3 attempts)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-03-27 09:45:59 -05:00
co-authored by Claude Opus 4.6
parent 6f0f92a9e4
commit b2cd54e941
15 changed files with 162 additions and 105 deletions
@@ -0,0 +1,23 @@
{
"id": "94a8bd16-6756-43af-b951-09a9e6b8300f",
"name": "clients",
"schema_fingerprint": "auto",
"objects": [
{
"bucket": "data",
"key": "datasets/clients.parquet",
"size_bytes": 21971,
"created_at": "2026-03-27T14:42:38.830329102Z"
}
],
"created_at": "2026-03-27T14:42:38.830331694Z",
"updated_at": "2026-03-27T14:42:38.830331694Z",
"description": "",
"owner": "",
"sensitivity": null,
"columns": [],
"lineage": null,
"freshness": null,
"tags": [],
"row_count": null
}