Phase E: Soft deletes (tombstones) for compliance-grade row deletion
Implements GDPR/CCPA-compatible row-level deletion without rewriting
the underlying Parquet. Tombstone markers live beside each dataset and
are applied at query time via a DataFusion view that excludes the
deleted row_key_values.
Schema (shared::types):
- Tombstone { dataset, row_key_column, row_key_value, deleted_at,
actor, reason }
- All tombstones for a dataset must share one row_key_column —
enforced at write so the query-time filter remains a single
WHERE NOT IN (...) clause
Storage (catalogd::tombstones):
- Per-dataset AppendLog at _catalog/tombstones/{dataset}/
- flush_threshold=1 + explicit flush after every append — tombstones
are high-value, low-frequency; durability on return is the contract
- Reuses storaged::append_log infra so compaction is already wired
(POST .../tombstones/compact will work once we expose it)
Catalog (catalogd::registry):
- add_tombstone validates dataset exists + key column compatibility
- list_tombstones for the GET endpoint
- TombstoneStore exposed via Registry::tombstones() for queryd
HTTP (catalogd::service):
- POST /catalog/datasets/by-name/{name}/tombstone
{ row_key_column, row_key_values[], actor, reason }
Returns rows_tombstoned count + per-value failure list (207 on
partial success).
- GET same path lists active tombstones with full audit info.
Query layer (queryd::context):
- Snapshot tombstones-by-dataset before registering tables
- Tombstoned tables: raw goes to "__raw__{name}", public "{name}"
becomes DataFusion view with
SELECT * FROM "__raw__{name}" WHERE CAST(col AS VARCHAR) NOT IN (...)
- CAST AS VARCHAR handles both string and integer key columns
- Untombstoned tables register as before — zero overhead
End-to-end on candidates (100K rows):
- Pick CAND-000001/2/3 (Linda/Charles/Kimberly)
- POST tombstone -> rows_tombstoned: 3
- COUNT(*) drops 100000 -> 99997
- WHERE candidate_id IN (those 3) -> 0 rows
- candidates_safe view transitively excludes them
(Linda+Denver: __raw__candidates=159, candidates_safe=158)
- Restart: COUNT still 99997, 3 tombstones reload from disk
Reversibility: tombstones are reversible deletes, not destruction.
Power users can still query "__raw__{name}" to see deleted rows.
Phase 13 access control is what stops a non-admin from accessing
__raw__* tables.
Limits / follow-up:
- Physical compaction not yet integrated — Phase 8's compact_files
doesn't read tombstones during merge. Tombstoned rows are still
on disk until that integration ships.
- Phase 9 journald event emission for tombstones not wired —
tombstone records carry their own actor+reason+timestamp so the
audit trail is intact, but cross-referencing with the mutation
event log would help compliance reporting.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -25,6 +25,8 @@ pub fn router(registry: Registry) -> Router {
|
||||
// Phase D: AI-safe views
|
||||
.route("/views", post(create_view).get(list_views))
|
||||
.route("/views/{name}", get(get_view).delete(delete_view))
|
||||
// Phase E: soft-delete tombstones
|
||||
.route("/datasets/by-name/{name}/tombstone", post(tombstone_rows).get(list_tombstones))
|
||||
.with_state(registry)
|
||||
}
|
||||
|
||||
@@ -269,3 +271,70 @@ async fn delete_view(
|
||||
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
|
||||
}
|
||||
}
|
||||
|
||||
// --- Phase E: soft-delete tombstones ---
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct TombstoneRequest {
|
||||
row_key_column: String,
|
||||
row_key_values: Vec<String>,
|
||||
#[serde(default)]
|
||||
actor: String,
|
||||
#[serde(default)]
|
||||
reason: String,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct TombstoneResponse {
|
||||
dataset: String,
|
||||
row_key_column: String,
|
||||
rows_tombstoned: usize,
|
||||
failures: Vec<String>,
|
||||
}
|
||||
|
||||
async fn tombstone_rows(
|
||||
State(registry): State<Registry>,
|
||||
Path(name): Path<String>,
|
||||
Json(req): Json<TombstoneRequest>,
|
||||
) -> impl IntoResponse {
|
||||
if req.row_key_values.is_empty() {
|
||||
return Err((StatusCode::BAD_REQUEST, "row_key_values is empty".to_string()));
|
||||
}
|
||||
|
||||
let mut ok = 0;
|
||||
let mut failures = Vec::new();
|
||||
for value in &req.row_key_values {
|
||||
match registry
|
||||
.add_tombstone(&name, &req.row_key_column, value, &req.actor, &req.reason)
|
||||
.await
|
||||
{
|
||||
Ok(_) => ok += 1,
|
||||
Err(e) => failures.push(format!("{value}: {e}")),
|
||||
}
|
||||
}
|
||||
|
||||
let status = if ok > 0 && failures.is_empty() {
|
||||
StatusCode::CREATED
|
||||
} else if ok > 0 {
|
||||
StatusCode::MULTI_STATUS
|
||||
} else {
|
||||
StatusCode::BAD_REQUEST
|
||||
};
|
||||
|
||||
Ok((status, Json(TombstoneResponse {
|
||||
dataset: name,
|
||||
row_key_column: req.row_key_column,
|
||||
rows_tombstoned: ok,
|
||||
failures,
|
||||
})))
|
||||
}
|
||||
|
||||
async fn list_tombstones(
|
||||
State(registry): State<Registry>,
|
||||
Path(name): Path<String>,
|
||||
) -> impl IntoResponse {
|
||||
match registry.list_tombstones(&name).await {
|
||||
Ok(ts) => Ok(Json(ts)),
|
||||
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user