Observer KB: new app for ChatGPT/Claude PDF knowledge extraction
- New branch: observer-kb - PRD: apps/observer-kb/docs/PRD.md - ChatGPT PDF parser: extracts conversation turns (user/assistant) - Upload portal: pure HTML/JS (no WASM), drag & drop, progress bars - Browse: view ingested datasets with preview - Search: semantic search across embedded conversations (HNSW) - Served on :3400 via systemd (observer-kb.service) - Uses Lakehouse core for storage, query, vectors, AI Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
@@ -0,0 +1,159 @@
|
||||
# PRD: Observer Knowledge Base
|
||||
|
||||
**Status:** Active
|
||||
**Created:** 2026-03-28
|
||||
**Owner:** J
|
||||
**Branch:** `observer-kb`
|
||||
**Parent:** Lakehouse core (crates/)
|
||||
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
AI conversations (ChatGPT, Claude, etc.) generate massive amounts of knowledge — decisions, research, code, strategies, troubleshooting — that disappear into PDF exports and chat logs. There's no way to:
|
||||
- Search across hundreds of conversations by meaning
|
||||
- Find "what did I discuss about X six months ago?"
|
||||
- Compare approaches across different conversations
|
||||
- Build a reusable knowledge base from past AI interactions
|
||||
- Track which ideas worked and which didn't
|
||||
|
||||
## Solution
|
||||
|
||||
An upload portal + knowledge extraction pipeline built on the Lakehouse infrastructure. Upload ChatGPT/Claude PDFs → extract conversations → chunk and embed → searchable knowledge base with comparison tools.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Upload Portal (web UI)
|
||||
│
|
||||
├── Upload large PDFs (chunked upload for big files)
|
||||
├── Progress tracking per file
|
||||
└── Visual extraction preview
|
||||
│
|
||||
▼
|
||||
Extraction Pipeline (uses Lakehouse ingestd)
|
||||
│
|
||||
├── PDF text extraction (lopdf)
|
||||
├── Conversation turn parsing (user/assistant)
|
||||
├── Topic detection
|
||||
├── Metadata extraction (dates, titles, participants)
|
||||
└── Quality scoring (how useful was this conversation?)
|
||||
│
|
||||
▼
|
||||
Lakehouse Storage
|
||||
│
|
||||
├── conversations table (id, title, source, date, turns, topic)
|
||||
├── turns table (conversation_id, turn_num, role, content)
|
||||
├── topics table (topic, conversation_ids, summary)
|
||||
└── vectors/ (embedded turns for semantic search)
|
||||
│
|
||||
▼
|
||||
Knowledge Interface
|
||||
│
|
||||
├── Search: "what did I discuss about Kubernetes?"
|
||||
├── Compare: conversation A vs B on same topic
|
||||
├── Timeline: when did I first explore this idea?
|
||||
├── Extract: pull key decisions/code/strategies
|
||||
└── Playbook: which approaches worked?
|
||||
```
|
||||
|
||||
## Tech Stack
|
||||
|
||||
Uses Lakehouse core — NOT a separate codebase:
|
||||
- **Storage:** Lakehouse storaged (Parquet on object storage)
|
||||
- **Catalog:** Lakehouse catalogd (metadata, PII, lineage)
|
||||
- **Query:** Lakehouse queryd (DataFusion SQL)
|
||||
- **Vectors:** Lakehouse vectord (HNSW search, RAG)
|
||||
- **AI:** Lakehouse aibridge → Ollama
|
||||
- **New:** Upload portal (standalone HTML/JS — NOT Dioxus WASM)
|
||||
|
||||
## Why HTML/JS for the Upload Portal
|
||||
|
||||
- No 39MB WASM payload
|
||||
- Chunked file upload (XMLHttpRequest with progress)
|
||||
- Works on any browser
|
||||
- Fast to iterate
|
||||
- The Dioxus UI can coexist for the SQL/Ask features
|
||||
|
||||
---
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 1: Upload Portal
|
||||
- Chunked file upload (handles 100MB+ PDFs)
|
||||
- Upload progress bar per file
|
||||
- Batch upload (drag & drop multiple)
|
||||
- File list with status (uploading, processing, done, failed)
|
||||
- Served as static HTML from the gateway
|
||||
|
||||
### Phase 2: Extraction Pipeline
|
||||
- ChatGPT PDF parser (conversation turns, titles, roles)
|
||||
- Claude PDF parser (similar structure)
|
||||
- Generic PDF fallback (page-based)
|
||||
- Extraction preview: see what was extracted before committing
|
||||
- Auto-embed extracted turns
|
||||
|
||||
### Phase 3: Knowledge Browser
|
||||
- Search across all conversations by meaning
|
||||
- Filter by date, topic, source
|
||||
- View individual conversations with turns
|
||||
- Highlight matching content in search results
|
||||
|
||||
### Phase 4: Comparison Tools
|
||||
- Side-by-side conversation comparison
|
||||
- Topic tracking across conversations over time
|
||||
- "Best approach" extraction from successful conversations
|
||||
- Playbook builder: curate successful patterns
|
||||
|
||||
---
|
||||
|
||||
## Data Model
|
||||
|
||||
```sql
|
||||
-- Conversations (one per PDF/chat export)
|
||||
conversations:
|
||||
conversation_id TEXT PRIMARY KEY
|
||||
title TEXT
|
||||
source TEXT -- "chatgpt", "claude", "other"
|
||||
source_file TEXT
|
||||
date TEXT
|
||||
turn_count INTEGER
|
||||
total_chars INTEGER
|
||||
topics TEXT -- pipe-separated
|
||||
summary TEXT -- AI-generated
|
||||
|
||||
-- Individual turns
|
||||
turns:
|
||||
turn_id TEXT PRIMARY KEY
|
||||
conversation_id TEXT
|
||||
turn_number INTEGER
|
||||
role TEXT -- "user", "assistant"
|
||||
content TEXT
|
||||
char_count INTEGER
|
||||
has_code BOOLEAN
|
||||
page_number INTEGER
|
||||
|
||||
-- Vector embeddings (via vectord)
|
||||
-- Stored in vectors/observer_kb.parquet
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Upload Requirements
|
||||
|
||||
- Max file size: 500MB per PDF
|
||||
- Chunked upload: 5MB chunks with resume on failure
|
||||
- Batch: up to 50 files at once
|
||||
- Progress: per-file percentage + overall
|
||||
- Dedup: skip already-ingested files (content hash)
|
||||
|
||||
---
|
||||
|
||||
## Non-Goals (for now)
|
||||
|
||||
- Live chat ingestion (only PDF exports)
|
||||
- Editing conversations
|
||||
- Multi-user access control
|
||||
- Real-time collaboration
|
||||
@@ -0,0 +1,178 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Observer KB</title>
|
||||
<style>
|
||||
:root{--bg:#0a0a0f;--surface:#12121a;--border:#2a2a3a;--text:#e0e0e8;--dim:#888898;--accent:#6c5ce7;--success:#00d2a0;--error:#ff6b6b;--mono:'JetBrains Mono','Fira Code',monospace}
|
||||
*{margin:0;padding:0;box-sizing:border-box}
|
||||
body{font-family:var(--mono);background:var(--bg);color:var(--text);font-size:13px;min-height:100vh}
|
||||
.app{max-width:1100px;margin:0 auto;padding:24px}
|
||||
header{display:flex;align-items:center;justify-content:space-between;padding:16px 0;border-bottom:1px solid var(--border);margin-bottom:24px}
|
||||
header h1{font-size:20px;color:var(--accent)}
|
||||
.stats{font-size:11px;color:var(--dim)}
|
||||
.tabs{display:flex;gap:2px;margin-bottom:24px}
|
||||
.tab{background:none;border:none;color:var(--dim);font-family:var(--mono);font-size:13px;padding:10px 20px;cursor:pointer;border-bottom:2px solid transparent}
|
||||
.tab:hover{color:var(--text)}.tab.active{color:var(--accent);border-bottom-color:var(--accent)}
|
||||
.upload-zone{border:2px dashed var(--border);border-radius:12px;padding:48px 24px;text-align:center;cursor:pointer;transition:all .2s;margin-bottom:24px}
|
||||
.upload-zone:hover,.upload-zone.dragover{border-color:var(--accent);background:rgba(108,92,231,.05)}
|
||||
.upload-zone h3{font-size:16px;margin-bottom:8px}
|
||||
.upload-zone p{color:var(--dim);font-size:12px}
|
||||
.file-list{display:flex;flex-direction:column;gap:8px}
|
||||
.file-item{background:var(--surface);border:1px solid var(--border);border-radius:8px;padding:14px 18px;display:grid;grid-template-columns:1fr 100px 120px;align-items:center;gap:12px}
|
||||
.file-name{font-weight:600;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}
|
||||
.file-size{color:var(--dim);font-size:11px;text-align:right}
|
||||
.file-status{font-size:11px;text-align:right}
|
||||
.uploading{color:var(--accent)}.done{color:var(--success)}.failed{color:var(--error)}
|
||||
.progress-bar{height:3px;background:var(--border);border-radius:2px;margin-top:6px;grid-column:1/-1}
|
||||
.progress-fill{height:100%;background:var(--accent);border-radius:2px;transition:width .3s}
|
||||
.search-bar{display:flex;gap:8px;margin-bottom:20px}
|
||||
.search-bar input{flex:1;background:var(--surface);border:2px solid var(--border);color:var(--text);font-family:var(--mono);font-size:14px;padding:12px 16px;border-radius:8px;outline:none}
|
||||
.search-bar input:focus{border-color:var(--accent)}
|
||||
.btn{background:var(--accent);color:white;border:none;padding:10px 20px;border-radius:6px;font-family:var(--mono);font-size:13px;cursor:pointer;font-weight:600}
|
||||
.btn:hover{opacity:.9}
|
||||
.ds-item{background:var(--surface);border:1px solid var(--border);border-radius:8px;padding:16px;margin-bottom:8px;cursor:pointer;transition:all .15s}
|
||||
.ds-item:hover{border-color:var(--accent)}
|
||||
.ds-title{font-weight:600;font-size:14px;margin-bottom:4px}
|
||||
.ds-meta{font-size:11px;color:var(--dim)}
|
||||
.hidden{display:none}
|
||||
.empty{color:var(--dim);text-align:center;padding:32px}
|
||||
.result-item{background:var(--surface);border:1px solid var(--border);border-radius:8px;padding:14px;margin-bottom:8px;border-left:3px solid var(--accent)}
|
||||
.result-score{font-size:11px;color:var(--accent);margin-bottom:4px}
|
||||
.result-text{font-size:12px;line-height:1.6;color:var(--dim)}
|
||||
#output{background:var(--surface);border:1px solid var(--border);border-radius:8px;padding:16px;margin-top:16px;font-size:12px;line-height:1.6;max-height:500px;overflow-y:auto;white-space:pre-wrap}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="app">
|
||||
<header><h1>OBSERVER KB</h1><div class="stats" id="stats">loading...</div></header>
|
||||
<div class="tabs">
|
||||
<button class="tab active" id="t-upload" onclick="showTab('upload')">Upload</button>
|
||||
<button class="tab" id="t-browse" onclick="showTab('browse')">Browse</button>
|
||||
<button class="tab" id="t-search" onclick="showTab('search')">Search</button>
|
||||
</div>
|
||||
|
||||
<div id="tab-upload">
|
||||
<div class="upload-zone" id="dropZone">
|
||||
<h3>Drop PDF files here</h3>
|
||||
<p>ChatGPT exports, Claude logs, any PDF — or click to browse</p>
|
||||
<input type="file" id="fileInput" multiple accept=".pdf,.txt,.csv,.json" style="display:none">
|
||||
</div>
|
||||
<div class="file-list" id="fileList"></div>
|
||||
</div>
|
||||
|
||||
<div id="tab-browse" class="hidden">
|
||||
<div id="dsList"></div>
|
||||
</div>
|
||||
|
||||
<div id="tab-search" class="hidden">
|
||||
<div class="search-bar">
|
||||
<input type="text" id="searchInput" placeholder="Search across all knowledge..." />
|
||||
<button class="btn" id="searchBtn">Search</button>
|
||||
</div>
|
||||
<div id="searchResults"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
const API=`http://${location.hostname}:3100`;
|
||||
const $=id=>document.getElementById(id);
|
||||
|
||||
function showTab(n){
|
||||
document.querySelectorAll('[id^="tab-"]').forEach(t=>t.classList.add('hidden'));
|
||||
$('tab-'+n).classList.remove('hidden');
|
||||
document.querySelectorAll('.tab').forEach(t=>t.classList.remove('active'));
|
||||
$('t-'+n).classList.add('active');
|
||||
if(n==='browse')loadDatasets();
|
||||
}
|
||||
|
||||
$('dropZone').onclick=()=>$('fileInput').click();
|
||||
$('dropZone').ondragover=e=>{e.preventDefault();$('dropZone').classList.add('dragover')};
|
||||
$('dropZone').ondragleave=()=>$('dropZone').classList.remove('dragover');
|
||||
$('dropZone').ondrop=e=>{e.preventDefault();$('dropZone').classList.remove('dragover');[...e.dataTransfer.files].forEach(uploadFile)};
|
||||
$('fileInput').onchange=e=>[...e.target.files].forEach(uploadFile);
|
||||
$('searchBtn').onclick=doSearch;
|
||||
$('searchInput').onkeydown=e=>{if(e.key==='Enter')doSearch()};
|
||||
|
||||
function uploadFile(file){
|
||||
const id='f'+Date.now()+Math.random().toString(36).substr(2,5);
|
||||
const item=document.createElement('div');
|
||||
item.className='file-item';
|
||||
item.id=id;
|
||||
const nameDiv=document.createElement('div');nameDiv.className='file-name';nameDiv.textContent=file.name;
|
||||
const sizeDiv=document.createElement('div');sizeDiv.className='file-size';sizeDiv.textContent=(file.size/1024/1024).toFixed(1)+' MB';
|
||||
const statusDiv=document.createElement('div');statusDiv.className='file-status uploading';statusDiv.textContent='uploading...';
|
||||
const progBar=document.createElement('div');progBar.className='progress-bar';
|
||||
const progFill=document.createElement('div');progFill.className='progress-fill';progFill.style.width='0%';
|
||||
progBar.appendChild(progFill);
|
||||
item.append(nameDiv,sizeDiv,statusDiv,progBar);
|
||||
$('fileList').prepend(item);
|
||||
|
||||
const fd=new FormData();fd.append('file',file);
|
||||
const xhr=new XMLHttpRequest();
|
||||
xhr.open('POST',API+'/ingest/file');
|
||||
xhr.upload.onprogress=e=>{if(e.lengthComputable){const p=Math.round(e.loaded/e.total*100);progFill.style.width=p+'%';statusDiv.textContent=p+'%'}};
|
||||
xhr.onload=()=>{
|
||||
if(xhr.status<300){statusDiv.className='file-status done';statusDiv.textContent='done';progFill.style.background='var(--success)';loadStats()}
|
||||
else{statusDiv.className='file-status failed';statusDiv.textContent='failed';progFill.style.background='var(--error)'}
|
||||
};
|
||||
xhr.onerror=()=>{statusDiv.className='file-status failed';statusDiv.textContent='error'};
|
||||
xhr.send(fd);
|
||||
}
|
||||
|
||||
async function loadStats(){
|
||||
try{const r=await(await fetch(API+'/catalog/datasets')).json();$('stats').textContent=r.length+' datasets'}
|
||||
catch{$('stats').textContent='offline'}
|
||||
}
|
||||
|
||||
async function loadDatasets(){
|
||||
try{
|
||||
const ds=await(await fetch(API+'/catalog/datasets')).json();
|
||||
const el=$('dsList');el.textContent='';
|
||||
if(!ds.length){el.textContent='No data yet';return}
|
||||
ds.forEach(d=>{
|
||||
const item=document.createElement('div');item.className='ds-item';
|
||||
const title=document.createElement('div');title.className='ds-title';title.textContent=d.name;
|
||||
const meta=document.createElement('div');meta.className='ds-meta';
|
||||
meta.textContent=`${d.row_count||'?'} rows · ${d.objects.length} file(s) · ${(d.sensitivity||'unclassified')}`;
|
||||
item.append(title,meta);
|
||||
item.onclick=()=>previewDS(d.name);
|
||||
el.appendChild(item);
|
||||
});
|
||||
}catch(e){$('dsList').textContent='Error: '+e}
|
||||
}
|
||||
|
||||
async function previewDS(name){
|
||||
try{
|
||||
const r=await(await fetch(API+'/query/sql',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({sql:`SELECT * FROM ${name} LIMIT 20`})})).json();
|
||||
const el=$('dsList');el.textContent='';
|
||||
const btn=document.createElement('button');btn.className='btn';btn.textContent='Back';btn.onclick=loadDatasets;
|
||||
const pre=document.createElement('div');pre.id='output';
|
||||
pre.textContent=JSON.stringify(r.rows,null,2);
|
||||
el.append(btn,pre);
|
||||
}catch(e){alert(e)}
|
||||
}
|
||||
|
||||
async function doSearch(){
|
||||
const q=$('searchInput').value;if(!q)return;
|
||||
const el=$('searchResults');el.textContent='searching...';
|
||||
try{
|
||||
const r=await fetch(API+'/vectors/hnsw/search',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({index_name:'observer_kb',query:q,top_k:10})});
|
||||
if(r.ok){
|
||||
const d=await r.json();el.textContent='';
|
||||
d.results.forEach(hit=>{
|
||||
const item=document.createElement('div');item.className='result-item';
|
||||
const score=document.createElement('div');score.className='result-score';score.textContent=`[${hit.score.toFixed(3)}] ${hit.source} / ${hit.doc_id}`;
|
||||
const text=document.createElement('div');text.className='result-text';text.textContent=hit.chunk_text.substring(0,300);
|
||||
item.append(score,text);el.appendChild(item);
|
||||
});
|
||||
if(!d.results.length)el.textContent='No results';
|
||||
}else{el.textContent='Vector index not built yet — upload PDFs first'}
|
||||
}catch(e){el.textContent='Error: '+e}
|
||||
}
|
||||
|
||||
loadStats();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user