profit/lakehouse
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

PRD v3: future-proofing roadmap — event journal, rich catalog, tool registry

Browse Source 
Phases 9-15 designed based on "future regret" analysis:
- Phase 9: Event journal (append-only mutation history — can't retrofit)
- Phase 10: Rich catalog v2 (ownership, sensitivity, lineage, freshness)
- Phase 11: Embedding versioning (model-proof vector layer)
- Phase 12: Tool registry (governed agent actions via MCP)
- Phase 13: Security & access control (field-level, row-level, audit)
- Phase 14: Schema evolution with AI migration rules
- Phase 15+: Federated query, DB connectors, OCR, fine-tuned models

8 design principles: store truth openly, describe richly, never destroy
evidence, secure centrally, expose through tools, version everything,
unstructured first-class, separate storage/compute/intelligence.

ADR-012 through ADR-016 documenting key future-proofing decisions.
Updated benchmarks: 2.47M rows, hot cache 9.8x speedup.
Updated operating rules: cheap-now/expensive-later built first.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-03-27 08:57:29 -05:00
co-authored by Claude Opus 4.6
parent 0b9da45647
commit 354c9c4a04
2 changed files with 193 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/PRD.md
+168 -28
View File
@@ -173,38 +173,147 @@ Make frequently-accessed data fast, and handle real-time updates without full re
- Compaction (moderate)
- Full ACID isolation (NOT attempting — single-writer model instead)
### Phase 9+: Future (not designed yet)
### Phase 8.5: Agent Workspaces ✅ COMPLETE
- Database connector ingest (PostgreSQL, MySQL, MSSQL → Parquet)
- PDF/document ingest (OCR → text → chunks → embed)
- Scheduled ingest (cron-based file watching)
- Multi-node query distribution
- Row-level access control
- Audit log (who queried what, when)
Per-contract overlays with daily/weekly/monthly tiers and instant handoff.
- WorkspaceManager with saved searches, shortlists, activity logs
- Zero-copy handoff between agents (pointer swap, not data copy)
- Persisted to object storage, rebuilt on startup
### Phase 9: Event Journal — Never Destroy Evidence
**Principle:** Every mutation is appended, never overwritten. This is the one decision that's impossible to retrofit — once history is lost, it's gone forever.
| Step | Deliverable | Gate |
|---|---|---|
| 9.1 | `journald` crate: append-only event log as Parquet | Every write/update/delete logged with who, when, what, old value, new value |
| 9.2 | Event schema: entity, field, old_value, new_value, actor, timestamp, source, workspace_id | Standardized across all mutations |
| 9.3 | Journal query: `SELECT * FROM journal WHERE entity = 'CAND-001' ORDER BY timestamp` | Full history of any record |
| 9.4 | Replay capability: rebuild any dataset's state at any point in time | Time-travel queries |
| 9.5 | Journal compaction: roll old events into monthly summary Parquet files | Prevent unbounded growth |
**Gate:** Change a candidate's phone number. Query shows the change. Journal shows old value, new value, who changed it, when, and why. Replay to yesterday's state.
**Why now:** In 3 years, compliance, AI auditability, and "why did the agent recommend this candidate" all require mutation history. Adding it later means you only have history from that day forward.
### Phase 10: Rich Catalog v2 — Metadata as Product
**Principle:** Every dataset should be self-describing. A new team member (or AI agent) should understand what data exists, who owns it, how fresh it is, and what's sensitive — without asking anyone.
| Step | Deliverable | Gate |
|---|---|---|
| 10.1 | Catalog schema upgrade: add owner, sensitivity, freshness_sla, description, tags, lineage | `GET /catalog/datasets` returns rich metadata |
| 10.2 | Sensitivity classification: PII, PHI, financial, public, internal | Sensitive fields tagged at ingest |
| 10.3 | Lineage tracking: source_system → ingest_job → dataset → derived_dataset | "Where did this data come from?" answerable |
| 10.4 | Freshness contracts: expected_update_frequency, last_updated, stale_after | Alert when data goes stale |
| 10.5 | Dataset contracts: required columns, type expectations, validation rules | Ingest rejects data that breaks the contract |
| 10.6 | Auto-documentation: AI generates dataset description from schema + sample data | New datasets self-describe on ingest |
**Gate:** Ingest a CSV. System auto-detects PII columns (email, phone, SSN patterns), tags them, generates a description, sets owner, and tracks lineage back to the source file.
**Why now:** Every dataset you ingest without metadata becomes a "mystery file" in 6 months. The metadata layer makes the difference between a searchable knowledge platform and a data graveyard.
### Phase 11: Embedding Versioning — Model-Proof Vector Layer
**Principle:** Embedding models will change. If you don't track which model created which vectors, upgrading means re-embedding everything from scratch.
| Step | Deliverable | Gate |
|---|---|---|
| 11.1 | Vector index metadata: model_name, model_version, dimensions, created_at | Every index knows its embedding model |
| 11.2 | Multi-version indexes: same data, different models, coexist | Search specifies which model version |
| 11.3 | Incremental re-embed: only new/changed docs get re-embedded on model upgrade | Model swap doesn't require full re-embed |
| 11.4 | A/B search: query both old and new model, compare results | Validate model upgrade before committing |
**Gate:** Upgrade from nomic-embed-text to a new model. Old index still works. New index builds incrementally. Compare search quality. Switch when ready.
### Phase 12: Tool Registry — Agent-Safe Business Actions
**Principle:** In 3 years, AI agents won't just query — they'll act. Instead of every agent getting raw SQL access, expose named, governed, audited business actions.
| Step | Deliverable | Gate |
|---|---|---|
| 12.1 | Tool definition: name, description, parameters, permissions, audit_level | `search_candidates(skills, city, min_years)` as a registered tool |
| 12.2 | Tool execution: validates params, checks permissions, logs usage, runs query | Agent calls tool, gets results, action is logged |
| 12.3 | Read vs write tools: read tools are permissive, write tools require confirmation | `get_candidate` = auto-approved, `update_phone` = requires review |
| 12.4 | MCP-compatible interface: expose tools via Model Context Protocol | Any MCP-compatible agent (Claude, GPT, local) can use them |
| 12.5 | Rate limiting + quotas per agent/tool | Prevent runaway agent from overwhelming the system |
**Gate:** An AI agent calls `search_candidates(skills="Python,AWS", city="Chicago", available=true)` → gets results → calls `shortlist_candidate(workspace_id, candidate_id, reason)` → action is logged, auditable, reversible.
**Why now:** The tool interface is cheap to build (it's just named endpoints with validation). But retrofitting audit logging and permission checks onto raw SQL access is a nightmare. Build the governed interface first.
### Phase 13: Security & Access Control
| Step | Deliverable | Gate |
|---|---|---|
| 13.1 | Field-level sensitivity tags (PII, PHI, financial) in catalog | Sensitive fields identified |
| 13.2 | Row-level access policies (agent A sees their candidates only) | Policy evaluated at query time |
| 13.3 | Column masking (show last 4 of SSN, redact salary for non-managers) | Masked results based on role |
| 13.4 | Query audit log (who queried what, when, which fields) | Every data access recorded |
| 13.5 | Policy-as-code (TOML/YAML rules, not hardcoded) | Non-engineer can update access rules |
### Phase 14: Schema Evolution + AI Migration
| Step | Deliverable | Gate |
|---|---|---|
| 14.1 | Schema diff detection: old schema vs new ingest → list changes | "Column renamed: first_name → full_name" |
| 14.2 | AI-generated migration rules: LLM suggests column mappings | "full_name = concat(first_name, ' ', last_name)" |
| 14.3 | Migration preview: show how old data maps to new schema before applying | Human approves before data transforms |
| 14.4 | Versioned schemas in catalog: v1, v2, v3 coexist | Queries specify version or use latest |
### Phase 15+: Horizon
- Federated multi-bucket query (client A's S3 + client B's S3 + yours)
- Database connector ingest (PostgreSQL, MySQL, MSSQL → Parquet via CDC)
- PDF OCR for scanned documents (Tesseract integration)
- Scheduled ingest (cron-based file watching, S3 event triggers)
- Specialized fine-tuned models per domain (staffing matcher, resume parser)
- Multi-node query distribution (DataFusion supports this architecturally)
- Video/audio transcript ingest + multimodal embeddings
---
## Reference Dataset: Staffing Company
Validated with realistic staffing company data:
Scale-tested on 128GB RAM server:
| Table | Rows | Description |
|---|---|---|
| candidates | 15,000 | Names, phones, emails, zip, skills, resume text, availability |
| clients | 500 | Companies, contacts, verticals, bill rates |
| job_orders | 3,000 | Positions with descriptions, requirements, rates |
| placements | 8,000 | Candidate↔job matches with dates, rates, recruiters |
| timesheets | 120,000 | Weekly hours, bill/pay totals, approvals |
| call_log | 80,000 | Phone CDR — who called whom, duration, disposition |
| email_log | 60,000 | Email tracking — subject, opened, direction |
| **Total** | **286,500** | **7 tables, cross-referenced** |
| Table | Rows | Size | Description |
|---|---|---|---|
| candidates | 100,000 | 10.1 MB | Names, phones, emails, zip, skills, resume text |
| clients | 2,000 | 33 KB | Companies, contacts, verticals |
| job_orders | 15,000 | 0.9 MB | Positions with descriptions, requirements, rates |
| placements | 50,000 | 1.2 MB | Candidate↔job matches with rates, recruiters |
| timesheets | 1,000,000 | 16.7 MB | Weekly hours, bill/pay totals, approvals |
| call_log | 800,000 | 34.3 MB | Phone CDR — who called whom, duration, disposition |
| email_log | 500,000 | 16.0 MB | Email tracking — subject, opened, direction |
| **Total** | **2,467,000** | **79 MB** | **7 tables, cross-referenced** |
Proven queries:
- Candidate search by skills + location + availability: 80ms
- Revenue by client with profit margins (JOIN 120K timesheets): 142ms
- Cold lead detection (candidates called 5+ times, never placed): 94ms
- Margin analysis by vertical (JOIN placements → job orders): 53ms
- Natural language → AI-generated SQL → execution → results: ~3s (model inference)
### Benchmarks (2.47M rows)
| Query | Cold (Parquet) | Hot (MemCache) | Speedup |
|---|---|---|---|
| 100K candidate filter (skills+city+status) | 257ms | 21ms | 12x |
| 1M timesheet aggregation + JOIN | 942ms | 96ms | 9.8x |
| 800K call log cross-reference (cold leads) | 642ms | — | — |
| Triple JOIN recruiter performance | 487ms | — | — |
| 500K email open rate aggregation | 259ms | — | — |
| COUNT all 2.47M rows | 84ms | — | — |
| 10K vector semantic search (cosine) | 450ms | — | — |
| Natural language → AI SQL → execute | ~3s | — | (model inference) |
### Vector Search
- 10K candidate resumes embedded in 204s (49 chunks/sec via Ollama)
- Semantic search over 10K vectors: ~450ms (brute-force cosine)
- RAG pipeline: question → embed → search → retrieve → LLM answer with citations
- AI correctly refuses to hallucinate when context doesn't support an answer
### Agent Workspaces
- Create per-contract workspace with saved searches + shortlists
- Instant handoff between agents — zero data copy
- Full activity timeline preserved across handoffs
---
@@ -233,16 +342,45 @@ Proven queries:
## Risks
### Technical Risks
| Risk | Severity | Mitigation |
|---|---|---|
| Vector search in Rust at scale | **High** | Start brute-force, evaluate `hora` crate, Qdrant as fallback |
| Incremental updates on Parquet | **High** | Delta files + merge-on-read, NOT full Delta Lake |
| Legacy data messiness | **High** | Conservative schema detection, default to string, user overrides |
| Schema evolution across ingests | **Medium** | Schema fingerprinting + versioned manifests |
| Memory pressure from hot cache | **Medium** | LRU eviction, configurable memory limit |
| 100K+ embedding timeout | **High** | Async background job with progress, not single HTTP request |
| Schema evolution across ingests | **Medium** | Schema fingerprinting + versioned manifests (Phase 14) |
| Memory pressure from hot cache | **Medium** | LRU eviction, configurable memory limit (tested: 408MB for 1.1M rows) |
| HNSW index persistence | **Medium** | Serialize alongside Parquet, rebuild on startup |
| Python sidecar as bottleneck | **Low** | Can replace with direct Ollama HTTP from Rust later |
### Strategic Risks (Future-Proofing)
| Risk | Impact | Phase |
|---|---|---|
| No mutation history → can't audit AI decisions | **Critical** — compliance, trust | Phase 9 (event journal) |
| No metadata → datasets become mystery files | **High** — onboarding, discovery | Phase 10 (rich catalog) |
| Embeddings locked to one model | **High** — can't upgrade models | Phase 11 (versioning) |
| Raw SQL as only interface → ungoverned agent access | **High** — security, auditability | Phase 12 (tool registry) |
| No sensitivity classification → compliance exposure | **Medium** — grows with data volume | Phase 13 (access control) |
| No schema evolution handling → ingest breaks on format change | **Medium** — grows with source count | Phase 14 (AI migration) |
---
## Design Principles (Future-Proofing)
These are the decisions that still look smart after the stack changes:
1. **Store the truth openly.** Parquet on object storage. No proprietary formats. Any engine can read it.
2. **Describe it richly.** Every dataset has an owner, lineage, sensitivity tags, freshness contract.
3. **Never destroy evidence.** Every mutation is journaled. Rebuild any state at any point in time.
4. **Secure it centrally.** Permissions live in the data layer, not application code.
5. **Expose it through reusable interfaces.** Named tools with contracts, not raw SQL for every consumer.
6. **Version everything.** Schemas, embeddings, models — all versioned, all coexist during migration.
7. **Make unstructured data first-class.** Every document gets: storage, text extraction, entity tags, chunks, embeddings, linkage.
8. **Separate storage from compute from intelligence.** Scale each independently. Replace any layer without touching the others.
---
## Operating Rules
@@ -253,5 +391,7 @@ Proven queries:
4. No silent architecture drift
5. Always work in smallest valid step
6. Always verify before moving on
7. **New:** Flag when something is genuinely hard vs just engineering work
8. **New:** If a phase reveals the approach is wrong, update the PRD before continuing
7. Flag when something is genuinely hard vs just engineering work
8. If a phase reveals the approach is wrong, update the PRD before continuing
9. **Cheap-now, expensive-later decisions get built first** (event journal, metadata, versioning)
10. **Build the governed interface before the raw interface** (tools before SQL for agents)