Phase D: AI-safe views — capability-surface projections over base data

Implements the llms3.com "AI-safe views" pattern: a named projection
that exposes only whitelisted columns, with optional row filter and
per-column redactions. AI agents (or Phase 13 roles) bind to the view;
they can never accidentally see PII even if they write raw SQL.

Schema (shared::types):
- AiView { name, base_dataset, columns: Vec<String>, row_filter,
           column_redactions: HashMap<String, Redaction>, ... }
- Redaction enum: Null | Hash | Mask { keep_prefix, keep_suffix }

Catalog (catalogd::registry):
- put_view validates base dataset exists + columns non-empty
- Persists JSON at _catalog/views/{name}.json (sanitized name)
- rebuild() loads views alongside dataset manifests on startup

Query layer (queryd::context):
- build_context registers every AiView as a DataFusion view object
- Constructed SELECT applies whitelist projection, WHERE filter, and
  redaction expressions per column
  - Mask: substr(prefix) + repeat('*', mid_len) + substr(suffix)
  - Hash: digest(value, 'sha256')
  - Null: CAST(NULL AS VARCHAR) AS col
- DataFusion handles JOINs/aggregates over the view natively — it's a
  real view, not a query rewrite

HTTP (catalogd::service):
- POST /catalog/views (create)
- GET  /catalog/views (list)
- GET  /catalog/views/{name} (full def)
- DELETE /catalog/views/{name}

End-to-end test on candidates (100K rows, 15 columns):

  candidates_safe view:
    columns: candidate_id, first_name, city, state, vertical,
             skills, years_experience, status
    row_filter: status != 'blocked'
    redaction: candidate_id mask(prefix=3, suffix=2)

  SELECT * FROM candidates_safe LIMIT 5
    -> 8 columns only, candidate_id shown as "CAN******01"
       (PII fields email/phone/last_name absent from result)

  SELECT email FROM candidates_safe
    -> fails (column not in projection)

  SELECT email FROM candidates
    -> succeeds (raw table still accessible by name —
       Phase 13 access control is the gate, not the view itself)

Survives restart — view definitions reload from object storage.

Limits / not in MVP:
- View CANNOT shadow base table by name (DataFusion treats them as
  separate identifiers; access control must restrict raw-table access)
- row_filter is treated as trusted SQL — operators must validate
  before persisting; only authenticated admin path should call put_view
- Redaction expressions assume column is castable to VARCHAR; numeric
  redactions could be misleading (a Hash on Int64 returns a hex string
  that won't equi-join with another hash on the same value type)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-16 09:16:44 -05:00
co-authored by Claude Opus 4.6
parent 24f1249a62
commit 09fd446c8d
5 changed files with 279 additions and 3 deletions
+56
View File
@@ -157,3 +157,59 @@ pub enum RefreshPolicy {
impl Default for RefreshPolicy {
fn default() -> Self { Self::Manual }
}
/// AI-safe view of a dataset (Phase D).
///
/// A view is a named projection over a base dataset that:
/// - whitelists the columns visible (everything else is invisible),
/// - optionally applies a row filter (a SQL WHERE clause fragment),
/// - optionally redacts column values (e.g. mask the last 4 of an SSN).
///
/// Queries that reference the view name see only the safe projection.
/// The base table remains separately accessible by its raw name — this
/// is a *capability surface*, not a data classification fence. Use
/// access control (Phase 13) on top to enforce who can query which view.
///
/// Stored as JSON at `_catalog/views/{name}.json` in the primary bucket
/// alongside dataset manifests. The view itself doesn't store data —
/// queryd materializes it as a DataFusion view at SessionContext build
/// time.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AiView {
pub name: String,
/// Source dataset name. Must exist in the catalog.
pub base_dataset: String,
/// Whitelisted columns. Empty list is invalid (would expose nothing).
pub columns: Vec<String>,
/// Optional WHERE clause fragment (without the `WHERE` keyword), e.g.
/// `status != 'blocked' AND city = 'Chicago'`. SQL injection here is
/// the operator's problem — only trust this from authenticated admins.
#[serde(default)]
pub row_filter: Option<String>,
/// Per-column redactions: `{column_name -> Redaction}`.
#[serde(default)]
pub column_redactions: std::collections::HashMap<String, Redaction>,
pub created_at: chrono::DateTime<chrono::Utc>,
#[serde(default)]
pub created_by: String,
#[serde(default)]
pub description: String,
}
/// How a column's values should be transformed before being returned.
/// `Mask` is the most common — keeps a few visible chars, replaces the
/// rest with `*`. `Hash` returns SHA-256 of the value for join keys you
/// want to preserve referential integrity on without exposing the value.
/// `Null` returns NULL unconditionally.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "snake_case")]
pub enum Redaction {
Null,
Hash,
Mask {
/// Number of leading characters left visible.
keep_prefix: usize,
/// Number of trailing characters left visible.
keep_suffix: usize,
},
}