Phase E: Soft deletes (tombstones) for compliance-grade row deletion

Implements GDPR/CCPA-compatible row-level deletion without rewriting
the underlying Parquet. Tombstone markers live beside each dataset and
are applied at query time via a DataFusion view that excludes the
deleted row_key_values.

Schema (shared::types):
- Tombstone { dataset, row_key_column, row_key_value, deleted_at,
              actor, reason }
- All tombstones for a dataset must share one row_key_column —
  enforced at write so the query-time filter remains a single
  WHERE NOT IN (...) clause

Storage (catalogd::tombstones):
- Per-dataset AppendLog at _catalog/tombstones/{dataset}/
- flush_threshold=1 + explicit flush after every append — tombstones
  are high-value, low-frequency; durability on return is the contract
- Reuses storaged::append_log infra so compaction is already wired
  (POST .../tombstones/compact will work once we expose it)

Catalog (catalogd::registry):
- add_tombstone validates dataset exists + key column compatibility
- list_tombstones for the GET endpoint
- TombstoneStore exposed via Registry::tombstones() for queryd

HTTP (catalogd::service):
- POST /catalog/datasets/by-name/{name}/tombstone
    { row_key_column, row_key_values[], actor, reason }
  Returns rows_tombstoned count + per-value failure list (207 on
  partial success).
- GET same path lists active tombstones with full audit info.

Query layer (queryd::context):
- Snapshot tombstones-by-dataset before registering tables
- Tombstoned tables: raw goes to "__raw__{name}", public "{name}"
  becomes DataFusion view with
  SELECT * FROM "__raw__{name}" WHERE CAST(col AS VARCHAR) NOT IN (...)
- CAST AS VARCHAR handles both string and integer key columns
- Untombstoned tables register as before — zero overhead

End-to-end on candidates (100K rows):
- Pick CAND-000001/2/3 (Linda/Charles/Kimberly)
- POST tombstone -> rows_tombstoned: 3
- COUNT(*) drops 100000 -> 99997
- WHERE candidate_id IN (those 3) -> 0 rows
- candidates_safe view transitively excludes them
  (Linda+Denver: __raw__candidates=159, candidates_safe=158)
- Restart: COUNT still 99997, 3 tombstones reload from disk

Reversibility: tombstones are reversible deletes, not destruction.
Power users can still query "__raw__{name}" to see deleted rows.
Phase 13 access control is what stops a non-admin from accessing
__raw__* tables.

Limits / follow-up:
- Physical compaction not yet integrated — Phase 8's compact_files
  doesn't read tombstones during merge. Tombstoned rows are still
  on disk until that integration ships.
- Phase 9 journald event emission for tombstones not wired —
  tombstone records carry their own actor+reason+timestamp so the
  audit trail is intact, but cross-referencing with the mutation
  event log would help compliance reporting.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-16 09:40:48 -05:00
co-authored by Claude Opus 4.6
parent 09fd446c8d
commit d87f2ccac6
7 changed files with 332 additions and 9 deletions
+29
View File
@@ -196,6 +196,35 @@ pub struct AiView {
pub description: String,
}
/// Soft-delete marker (Phase E).
///
/// Tombstones live beside the dataset in `_catalog/tombstones/{dataset}/`
/// as append-log JSONL. Query-time filter in queryd reads all tombstones
/// for each dataset and wraps the base table in a DataFusion view that
/// excludes tombstoned rows. Physical deletion happens later (compaction),
/// so the row count immediately reflects the delete but data is still on
/// disk until compact runs. That's deliberate — it gives a reversal
/// window and keeps the event journal audit trail intact.
///
/// All tombstones for a given dataset must use the same `row_key_column`
/// (enforced on write); otherwise the query-time filter can't be built
/// as a single WHERE clause.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Tombstone {
pub dataset: String,
/// Column name that identifies the row (e.g. "candidate_id").
pub row_key_column: String,
/// Value of that column for the tombstoned row.
pub row_key_value: String,
pub deleted_at: chrono::DateTime<chrono::Utc>,
/// Human / system actor responsible for the delete (audit).
#[serde(default)]
pub actor: String,
/// Why (e.g. "GDPR request #1234", "user-requested erasure").
#[serde(default)]
pub reason: String,
}
/// How a column's values should be transformed before being returned.
/// `Mask` is the most common — keeps a few visible chars, replaces the
/// rest with `*`. `Hash` returns SHA-256 of the value for join keys you