# PRD: Observer Knowledge Base

**Status:** Active
**Created:** 2026-03-28
**Owner:** J
**Branch:** `observer-kb`
**Parent:** Lakehouse core (crates/)

---

## Problem

AI conversations (ChatGPT, Claude, etc.) generate massive amounts of knowledge — decisions, research, code, strategies, troubleshooting — that disappear into PDF exports and chat logs. There's no way to:
- Search across hundreds of conversations by meaning
- Find "what did I discuss about X six months ago?"
- Compare approaches across different conversations
- Build a reusable knowledge base from past AI interactions
- Track which ideas worked and which didn't

## Solution

An upload portal + knowledge extraction pipeline built on the Lakehouse infrastructure. Upload ChatGPT/Claude PDFs → extract conversations → chunk and embed → searchable knowledge base with comparison tools.

---

## Architecture

```
Upload Portal (web UI)
    │
    ├── Upload large PDFs (chunked upload for big files)
    ├── Progress tracking per file
    └── Visual extraction preview
    │
    ▼
Extraction Pipeline (uses Lakehouse ingestd)
    │
    ├── PDF text extraction (lopdf)
    ├── Conversation turn parsing (user/assistant)
    ├── Topic detection
    ├── Metadata extraction (dates, titles, participants)
    └── Quality scoring (how useful was this conversation?)
    │
    ▼
Lakehouse Storage
    │
    ├── conversations table (id, title, source, date, turns, topic)
    ├── turns table (conversation_id, turn_num, role, content)
    ├── topics table (topic, conversation_ids, summary)
    └── vectors/ (embedded turns for semantic search)
    │
    ▼
Knowledge Interface
    │
    ├── Search: "what did I discuss about Kubernetes?"
    ├── Compare: conversation A vs B on same topic
    ├── Timeline: when did I first explore this idea?
    ├── Extract: pull key decisions/code/strategies
    └── Playbook: which approaches worked?
```

## Tech Stack

Uses Lakehouse core — NOT a separate codebase:
- **Storage:** Lakehouse storaged (Parquet on object storage)
- **Catalog:** Lakehouse catalogd (metadata, PII, lineage)
- **Query:** Lakehouse queryd (DataFusion SQL)
- **Vectors:** Lakehouse vectord (HNSW search, RAG)
- **AI:** Lakehouse aibridge → Ollama
- **New:** Upload portal (standalone HTML/JS — NOT Dioxus WASM)

## Why HTML/JS for the Upload Portal

- No 39MB WASM payload
- Chunked file upload (XMLHttpRequest with progress)
- Works on any browser
- Fast to iterate
- The Dioxus UI can coexist for the SQL/Ask features

---

## Phases

### Phase 1: Upload Portal
- Chunked file upload (handles 100MB+ PDFs)
- Upload progress bar per file
- Batch upload (drag & drop multiple)
- File list with status (uploading, processing, done, failed)
- Served as static HTML from the gateway

### Phase 2: Extraction Pipeline
- ChatGPT PDF parser (conversation turns, titles, roles)
- Claude PDF parser (similar structure)
- Generic PDF fallback (page-based)
- Extraction preview: see what was extracted before committing
- Auto-embed extracted turns

### Phase 3: Knowledge Browser
- Search across all conversations by meaning
- Filter by date, topic, source
- View individual conversations with turns
- Highlight matching content in search results

### Phase 4: Comparison Tools
- Side-by-side conversation comparison
- Topic tracking across conversations over time
- "Best approach" extraction from successful conversations
- Playbook builder: curate successful patterns

---

## Data Model

```sql
-- Conversations (one per PDF/chat export)
conversations:
  conversation_id TEXT PRIMARY KEY
  title TEXT
  source TEXT              -- "chatgpt", "claude", "other"
  source_file TEXT
  date TEXT
  turn_count INTEGER
  total_chars INTEGER
  topics TEXT              -- pipe-separated
  summary TEXT             -- AI-generated

-- Individual turns
turns:
  turn_id TEXT PRIMARY KEY
  conversation_id TEXT
  turn_number INTEGER
  role TEXT                -- "user", "assistant"
  content TEXT
  char_count INTEGER
  has_code BOOLEAN
  page_number INTEGER

-- Vector embeddings (via vectord)
-- Stored in vectors/observer_kb.parquet
```

---

## Upload Requirements

- Max file size: 500MB per PDF
- Chunked upload: 5MB chunks with resume on failure
- Batch: up to 50 files at once
- Progress: per-file percentage + overall
- Dedup: skip already-ingested files (content hash)

---

## Non-Goals (for now)

- Live chat ingestion (only PDF exports)
- Editing conversations
- Multi-user access control
- Real-time collaboration