rootandClaude Opus 4.6
c49b30053b
- New branch: observer-kb - PRD: apps/observer-kb/docs/PRD.md - ChatGPT PDF parser: extracts conversation turns (user/assistant) - Upload portal: pure HTML/JS (no WASM), drag & drop, progress bars - Browse: view ingested datasets with preview - Search: semantic search across embedded conversations (HNSW) - Served on :3400 via systemd (observer-kb.service) - Uses Lakehouse core for storage, query, vectors, AI Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
160 lines
4.5 KiB
Markdown
160 lines
4.5 KiB
Markdown
# PRD: Observer Knowledge Base
|
|
|
|
**Status:** Active
|
|
**Created:** 2026-03-28
|
|
**Owner:** J
|
|
**Branch:** `observer-kb`
|
|
**Parent:** Lakehouse core (crates/)
|
|
|
|
---
|
|
|
|
## Problem
|
|
|
|
AI conversations (ChatGPT, Claude, etc.) generate massive amounts of knowledge — decisions, research, code, strategies, troubleshooting — that disappear into PDF exports and chat logs. There's no way to:
|
|
- Search across hundreds of conversations by meaning
|
|
- Find "what did I discuss about X six months ago?"
|
|
- Compare approaches across different conversations
|
|
- Build a reusable knowledge base from past AI interactions
|
|
- Track which ideas worked and which didn't
|
|
|
|
## Solution
|
|
|
|
An upload portal + knowledge extraction pipeline built on the Lakehouse infrastructure. Upload ChatGPT/Claude PDFs → extract conversations → chunk and embed → searchable knowledge base with comparison tools.
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Upload Portal (web UI)
|
|
│
|
|
├── Upload large PDFs (chunked upload for big files)
|
|
├── Progress tracking per file
|
|
└── Visual extraction preview
|
|
│
|
|
▼
|
|
Extraction Pipeline (uses Lakehouse ingestd)
|
|
│
|
|
├── PDF text extraction (lopdf)
|
|
├── Conversation turn parsing (user/assistant)
|
|
├── Topic detection
|
|
├── Metadata extraction (dates, titles, participants)
|
|
└── Quality scoring (how useful was this conversation?)
|
|
│
|
|
▼
|
|
Lakehouse Storage
|
|
│
|
|
├── conversations table (id, title, source, date, turns, topic)
|
|
├── turns table (conversation_id, turn_num, role, content)
|
|
├── topics table (topic, conversation_ids, summary)
|
|
└── vectors/ (embedded turns for semantic search)
|
|
│
|
|
▼
|
|
Knowledge Interface
|
|
│
|
|
├── Search: "what did I discuss about Kubernetes?"
|
|
├── Compare: conversation A vs B on same topic
|
|
├── Timeline: when did I first explore this idea?
|
|
├── Extract: pull key decisions/code/strategies
|
|
└── Playbook: which approaches worked?
|
|
```
|
|
|
|
## Tech Stack
|
|
|
|
Uses Lakehouse core — NOT a separate codebase:
|
|
- **Storage:** Lakehouse storaged (Parquet on object storage)
|
|
- **Catalog:** Lakehouse catalogd (metadata, PII, lineage)
|
|
- **Query:** Lakehouse queryd (DataFusion SQL)
|
|
- **Vectors:** Lakehouse vectord (HNSW search, RAG)
|
|
- **AI:** Lakehouse aibridge → Ollama
|
|
- **New:** Upload portal (standalone HTML/JS — NOT Dioxus WASM)
|
|
|
|
## Why HTML/JS for the Upload Portal
|
|
|
|
- No 39MB WASM payload
|
|
- Chunked file upload (XMLHttpRequest with progress)
|
|
- Works on any browser
|
|
- Fast to iterate
|
|
- The Dioxus UI can coexist for the SQL/Ask features
|
|
|
|
---
|
|
|
|
## Phases
|
|
|
|
### Phase 1: Upload Portal
|
|
- Chunked file upload (handles 100MB+ PDFs)
|
|
- Upload progress bar per file
|
|
- Batch upload (drag & drop multiple)
|
|
- File list with status (uploading, processing, done, failed)
|
|
- Served as static HTML from the gateway
|
|
|
|
### Phase 2: Extraction Pipeline
|
|
- ChatGPT PDF parser (conversation turns, titles, roles)
|
|
- Claude PDF parser (similar structure)
|
|
- Generic PDF fallback (page-based)
|
|
- Extraction preview: see what was extracted before committing
|
|
- Auto-embed extracted turns
|
|
|
|
### Phase 3: Knowledge Browser
|
|
- Search across all conversations by meaning
|
|
- Filter by date, topic, source
|
|
- View individual conversations with turns
|
|
- Highlight matching content in search results
|
|
|
|
### Phase 4: Comparison Tools
|
|
- Side-by-side conversation comparison
|
|
- Topic tracking across conversations over time
|
|
- "Best approach" extraction from successful conversations
|
|
- Playbook builder: curate successful patterns
|
|
|
|
---
|
|
|
|
## Data Model
|
|
|
|
```sql
|
|
-- Conversations (one per PDF/chat export)
|
|
conversations:
|
|
conversation_id TEXT PRIMARY KEY
|
|
title TEXT
|
|
source TEXT -- "chatgpt", "claude", "other"
|
|
source_file TEXT
|
|
date TEXT
|
|
turn_count INTEGER
|
|
total_chars INTEGER
|
|
topics TEXT -- pipe-separated
|
|
summary TEXT -- AI-generated
|
|
|
|
-- Individual turns
|
|
turns:
|
|
turn_id TEXT PRIMARY KEY
|
|
conversation_id TEXT
|
|
turn_number INTEGER
|
|
role TEXT -- "user", "assistant"
|
|
content TEXT
|
|
char_count INTEGER
|
|
has_code BOOLEAN
|
|
page_number INTEGER
|
|
|
|
-- Vector embeddings (via vectord)
|
|
-- Stored in vectors/observer_kb.parquet
|
|
```
|
|
|
|
---
|
|
|
|
## Upload Requirements
|
|
|
|
- Max file size: 500MB per PDF
|
|
- Chunked upload: 5MB chunks with resume on failure
|
|
- Batch: up to 50 files at once
|
|
- Progress: per-file percentage + overall
|
|
- Dedup: skip already-ingested files (content hash)
|
|
|
|
---
|
|
|
|
## Non-Goals (for now)
|
|
|
|
- Live chat ingestion (only PDF exports)
|
|
- Editing conversations
|
|
- Multi-user access control
|
|
- Real-time collaboration
|