lakehouse/apps/observer-kb/docs/PRD.md

PRD: Observer Knowledge Base

Status: Active Created: 2026-03-28 Owner: J Branch: observer-kb Parent: Lakehouse core (crates/)

---

Problem

AI conversations (ChatGPT, Claude, etc.) generate massive amounts of knowledge — decisions, research, code, strategies, troubleshooting — that disappear into PDF exports and chat logs. There's no way to:

• Search across hundreds of conversations by meaning
• Find "what did I discuss about X six months ago?"
• Compare approaches across different conversations
• Build a reusable knowledge base from past AI interactions
• Track which ideas worked and which didn't

Solution

An upload portal + knowledge extraction pipeline built on the Lakehouse infrastructure. Upload ChatGPT/Claude PDFs → extract conversations → chunk and embed → searchable knowledge base with comparison tools.

---

Architecture

Upload Portal (web UI)
    │
    ├── Upload large PDFs (chunked upload for big files)
    ├── Progress tracking per file
    └── Visual extraction preview
    │
    ▼
Extraction Pipeline (uses Lakehouse ingestd)
    │
    ├── PDF text extraction (lopdf)
    ├── Conversation turn parsing (user/assistant)
    ├── Topic detection
    ├── Metadata extraction (dates, titles, participants)
    └── Quality scoring (how useful was this conversation?)
    │
    ▼
Lakehouse Storage
    │
    ├── conversations table (id, title, source, date, turns, topic)
    ├── turns table (conversation_id, turn_num, role, content)
    ├── topics table (topic, conversation_ids, summary)
    └── vectors/ (embedded turns for semantic search)
    │
    ▼
Knowledge Interface
    │
    ├── Search: "what did I discuss about Kubernetes?"
    ├── Compare: conversation A vs B on same topic
    ├── Timeline: when did I first explore this idea?
    ├── Extract: pull key decisions/code/strategies
    └── Playbook: which approaches worked?

Tech Stack

Uses Lakehouse core — NOT a separate codebase:

• Storage: Lakehouse storaged (Parquet on object storage)
• Catalog: Lakehouse catalogd (metadata, PII, lineage)
• Query: Lakehouse queryd (DataFusion SQL)
• Vectors: Lakehouse vectord (HNSW search, RAG)
• AI: Lakehouse aibridge → Ollama
• New: Upload portal (standalone HTML/JS — NOT Dioxus WASM)

Why HTML/JS for the Upload Portal

• No 39MB WASM payload
• Chunked file upload (XMLHttpRequest with progress)
• Works on any browser
• Fast to iterate
• The Dioxus UI can coexist for the SQL/Ask features

---

Phases

Phase 1: Upload Portal

• Chunked file upload (handles 100MB+ PDFs)
• Upload progress bar per file
• Batch upload (drag & drop multiple)
• File list with status (uploading, processing, done, failed)
• Served as static HTML from the gateway

Phase 2: Extraction Pipeline

• ChatGPT PDF parser (conversation turns, titles, roles)
• Claude PDF parser (similar structure)
• Generic PDF fallback (page-based)
• Extraction preview: see what was extracted before committing
• Auto-embed extracted turns

Phase 3: Knowledge Browser

• Search across all conversations by meaning
• Filter by date, topic, source
• View individual conversations with turns
• Highlight matching content in search results

Phase 4: Comparison Tools

• Side-by-side conversation comparison
• Topic tracking across conversations over time
• "Best approach" extraction from successful conversations
• Playbook builder: curate successful patterns

---

Data Model

-- Conversations (one per PDF/chat export)
conversations:
  conversation_id TEXT PRIMARY KEY
  title TEXT
  source TEXT              -- "chatgpt", "claude", "other"
  source_file TEXT
  date TEXT
  turn_count INTEGER
  total_chars INTEGER
  topics TEXT              -- pipe-separated
  summary TEXT             -- AI-generated

-- Individual turns
turns:
  turn_id TEXT PRIMARY KEY
  conversation_id TEXT
  turn_number INTEGER
  role TEXT                -- "user", "assistant"
  content TEXT
  char_count INTEGER
  has_code BOOLEAN
  page_number INTEGER

-- Vector embeddings (via vectord)
-- Stored in vectors/observer_kb.parquet

---

Upload Requirements

• Max file size: 500MB per PDF
• Chunked upload: 5MB chunks with resume on failure
• Batch: up to 50 files at once
• Progress: per-file percentage + overall
• Dedup: skip already-ingested files (content hash)

---

Non-Goals (for now)

• Live chat ingestion (only PDF exports)
• Editing conversations
• Multi-user access control
• Real-time collaboration