Observer KB: new app for ChatGPT/Claude PDF knowledge extraction

- New branch: observer-kb - PRD: apps/observer-kb/docs/PRD.md - ChatGPT PDF parser: extracts conversation turns (user/assistant) - Upload portal: pure HTML/JS (no WASM), drag & drop, progress bars - Browse: view ingested datasets with preview - Search: semantic search across embedded conversations (HNSW) - Served on :3400 via systemd (observer-kb.service) - Uses Lakehouse core for storage, query, vectors, AI Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 22:23:09 -05:00
parent 84407eeb51
commit c49b30053b
5 changed files with 580 additions and 1 deletions
@@ -0,0 +1,159 @@
+# PRD: Observer Knowledge Base
+
+**Status:** Active
+**Created:** 2026-03-28
+**Owner:** J
+**Branch:** `observer-kb`
+**Parent:** Lakehouse core (crates/)
+
+---
+
+## Problem
+
+AI conversations (ChatGPT, Claude, etc.) generate massive amounts of knowledge — decisions, research, code, strategies, troubleshooting — that disappear into PDF exports and chat logs. There's no way to:
+- Search across hundreds of conversations by meaning
+- Find "what did I discuss about X six months ago?"
+- Compare approaches across different conversations
+- Build a reusable knowledge base from past AI interactions
+- Track which ideas worked and which didn't
+
+## Solution
+
+An upload portal + knowledge extraction pipeline built on the Lakehouse infrastructure. Upload ChatGPT/Claude PDFs → extract conversations → chunk and embed → searchable knowledge base with comparison tools.
+
+---
+
+## Architecture
+
+```
+Upload Portal (web UI)
+    │
+    ├── Upload large PDFs (chunked upload for big files)
+    ├── Progress tracking per file
+    └── Visual extraction preview
+    │
+    ▼
+Extraction Pipeline (uses Lakehouse ingestd)
+    │
+    ├── PDF text extraction (lopdf)
+    ├── Conversation turn parsing (user/assistant)
+    ├── Topic detection
+    ├── Metadata extraction (dates, titles, participants)
+    └── Quality scoring (how useful was this conversation?)
+    │
+    ▼
+Lakehouse Storage
+    │
+    ├── conversations table (id, title, source, date, turns, topic)
+    ├── turns table (conversation_id, turn_num, role, content)
+    ├── topics table (topic, conversation_ids, summary)
+    └── vectors/ (embedded turns for semantic search)
+    │
+    ▼
+Knowledge Interface
+    │
+    ├── Search: "what did I discuss about Kubernetes?"
+    ├── Compare: conversation A vs B on same topic
+    ├── Timeline: when did I first explore this idea?
+    ├── Extract: pull key decisions/code/strategies
+    └── Playbook: which approaches worked?
+```
+
+## Tech Stack
+
+Uses Lakehouse core — NOT a separate codebase:
+- **Storage:** Lakehouse storaged (Parquet on object storage)
+- **Catalog:** Lakehouse catalogd (metadata, PII, lineage)
+- **Query:** Lakehouse queryd (DataFusion SQL)
+- **Vectors:** Lakehouse vectord (HNSW search, RAG)
+- **AI:** Lakehouse aibridge → Ollama
+- **New:** Upload portal (standalone HTML/JS — NOT Dioxus WASM)
+
+## Why HTML/JS for the Upload Portal
+
+- No 39MB WASM payload
+- Chunked file upload (XMLHttpRequest with progress)
+- Works on any browser
+- Fast to iterate
+- The Dioxus UI can coexist for the SQL/Ask features
+
+---
+
+## Phases
+
+### Phase 1: Upload Portal
+- Chunked file upload (handles 100MB+ PDFs)
+- Upload progress bar per file
+- Batch upload (drag & drop multiple)
+- File list with status (uploading, processing, done, failed)
+- Served as static HTML from the gateway
+
+### Phase 2: Extraction Pipeline
+- ChatGPT PDF parser (conversation turns, titles, roles)
+- Claude PDF parser (similar structure)
+- Generic PDF fallback (page-based)
+- Extraction preview: see what was extracted before committing
+- Auto-embed extracted turns
+
+### Phase 3: Knowledge Browser
+- Search across all conversations by meaning
+- Filter by date, topic, source
+- View individual conversations with turns
+- Highlight matching content in search results
+
+### Phase 4: Comparison Tools
+- Side-by-side conversation comparison
+- Topic tracking across conversations over time
+- "Best approach" extraction from successful conversations
+- Playbook builder: curate successful patterns
+
+---
+
+## Data Model
+
+```sql
+-- Conversations (one per PDF/chat export)
+conversations:
+  conversation_id TEXT PRIMARY KEY
+  title TEXT
+  source TEXT              -- "chatgpt", "claude", "other"
+  source_file TEXT
+  date TEXT
+  turn_count INTEGER
+  total_chars INTEGER
+  topics TEXT              -- pipe-separated
+  summary TEXT             -- AI-generated
+
+-- Individual turns
+turns:
+  turn_id TEXT PRIMARY KEY
+  conversation_id TEXT
+  turn_number INTEGER
+  role TEXT                -- "user", "assistant"
+  content TEXT
+  char_count INTEGER
+  has_code BOOLEAN
+  page_number INTEGER
+
+-- Vector embeddings (via vectord)
+-- Stored in vectors/observer_kb.parquet
+```
+
+---
+
+## Upload Requirements
+
+- Max file size: 500MB per PDF
+- Chunked upload: 5MB chunks with resume on failure
+- Batch: up to 50 files at once
+- Progress: per-file percentage + overall
+- Dedup: skip already-ingested files (content hash)
+
+---
+
+## Non-Goals (for now)
+
+- Live chat ingestion (only PDF exports)
+- Editing conversations
+- Multi-user access control
+- Real-time collaboration