Phase E: Scheduled ingest — the substrate runs itself

Background Scheduler task fires due ingests on interval, records
outcomes, reschedules. Single-flight per schedule_id so a slow run
can't pile up. 10s tick cadence, schedules' own intervals independent.

ScheduleDef persisted as JSON at primary://_schedules/{id}.json,
rebuilt on startup. ScheduleKind supports Mysql and Postgres (both
through existing streaming paths). ScheduleTrigger::Interval is
live; Cron variant defined in the enum but parsing stubbed with a
safe 1h fallback.

next_run_at set to "now" on creation so operators see success or
failure within one tick — no waiting for the first full interval.
run-now endpoint fires even when schedule is disabled (manual
override for testing). Full catalog integration: PII detection,
lineage with redacted DSN, mark-stale + autotune agent trigger.

Verified live: 20s MySQL schedule against MariaDB lh_demo.customers.
Source mutated between runs (added row + updated value). Second
auto-fire picked up both changes (10→11 rows). DataFusion SQL
confirmed mutations in the lakehouse. 6 unit tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-16 20:36:04 -05:00
co-authored by Claude Opus 4.6
parent 0d037cfac1
commit 7c1222d240
4 changed files with 799 additions and 3 deletions
+22
View File
@@ -118,12 +118,34 @@ async fn main() {
.nest("/catalog", catalogd::service::router(registry.clone()))
.nest("/query", queryd::service::router(engine.clone()))
.nest("/ai", aibridge::service::router(ai_client.clone()))
;
// Phase E: scheduled ingest
let sched_store = ingestd::schedule::ScheduleStore::new(store.clone());
if let Ok(n) = sched_store.rebuild().await {
if n > 0 { tracing::info!("rebuilt {n} persisted schedule(s)"); }
}
let scheduler = ingestd::schedule::Scheduler {
store: sched_store.clone(),
ingest: ingestd::schedule::SchedulerIngestDeps {
store: store.clone(),
registry: registry.clone(),
buckets: bucket_registry.clone(),
agent_handle: Some(agent_handle.clone()),
index_registry: index_reg.clone(),
},
tick_secs: 10,
};
scheduler.spawn();
app = app
.nest("/ingest", ingestd::service::router(ingestd::service::IngestState {
store: store.clone(),
registry: registry.clone(),
buckets: bucket_registry.clone(),
agent_handle: agent_handle.clone(),
index_registry: index_reg.clone(),
schedules: sched_store,
}))
.nest("/vectors", vectord::service::router(vectord::service::VectorState {
store: store.clone(),