Observer KB: new app for ChatGPT/Claude PDF knowledge extraction
- New branch: observer-kb - PRD: apps/observer-kb/docs/PRD.md - ChatGPT PDF parser: extracts conversation turns (user/assistant) - Upload portal: pure HTML/JS (no WASM), drag & drop, progress bars - Browse: view ingested datasets with preview - Search: semantic search across embedded conversations (HNSW) - Served on :3400 via systemd (observer-kb.service) - Uses Lakehouse core for storage, query, vectors, AI Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
@@ -0,0 +1,159 @@
|
|||||||
|
# PRD: Observer Knowledge Base
|
||||||
|
|
||||||
|
**Status:** Active
|
||||||
|
**Created:** 2026-03-28
|
||||||
|
**Owner:** J
|
||||||
|
**Branch:** `observer-kb`
|
||||||
|
**Parent:** Lakehouse core (crates/)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
AI conversations (ChatGPT, Claude, etc.) generate massive amounts of knowledge — decisions, research, code, strategies, troubleshooting — that disappear into PDF exports and chat logs. There's no way to:
|
||||||
|
- Search across hundreds of conversations by meaning
|
||||||
|
- Find "what did I discuss about X six months ago?"
|
||||||
|
- Compare approaches across different conversations
|
||||||
|
- Build a reusable knowledge base from past AI interactions
|
||||||
|
- Track which ideas worked and which didn't
|
||||||
|
|
||||||
|
## Solution
|
||||||
|
|
||||||
|
An upload portal + knowledge extraction pipeline built on the Lakehouse infrastructure. Upload ChatGPT/Claude PDFs → extract conversations → chunk and embed → searchable knowledge base with comparison tools.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
Upload Portal (web UI)
|
||||||
|
│
|
||||||
|
├── Upload large PDFs (chunked upload for big files)
|
||||||
|
├── Progress tracking per file
|
||||||
|
└── Visual extraction preview
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Extraction Pipeline (uses Lakehouse ingestd)
|
||||||
|
│
|
||||||
|
├── PDF text extraction (lopdf)
|
||||||
|
├── Conversation turn parsing (user/assistant)
|
||||||
|
├── Topic detection
|
||||||
|
├── Metadata extraction (dates, titles, participants)
|
||||||
|
└── Quality scoring (how useful was this conversation?)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Lakehouse Storage
|
||||||
|
│
|
||||||
|
├── conversations table (id, title, source, date, turns, topic)
|
||||||
|
├── turns table (conversation_id, turn_num, role, content)
|
||||||
|
├── topics table (topic, conversation_ids, summary)
|
||||||
|
└── vectors/ (embedded turns for semantic search)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Knowledge Interface
|
||||||
|
│
|
||||||
|
├── Search: "what did I discuss about Kubernetes?"
|
||||||
|
├── Compare: conversation A vs B on same topic
|
||||||
|
├── Timeline: when did I first explore this idea?
|
||||||
|
├── Extract: pull key decisions/code/strategies
|
||||||
|
└── Playbook: which approaches worked?
|
||||||
|
```
|
||||||
|
|
||||||
|
## Tech Stack
|
||||||
|
|
||||||
|
Uses Lakehouse core — NOT a separate codebase:
|
||||||
|
- **Storage:** Lakehouse storaged (Parquet on object storage)
|
||||||
|
- **Catalog:** Lakehouse catalogd (metadata, PII, lineage)
|
||||||
|
- **Query:** Lakehouse queryd (DataFusion SQL)
|
||||||
|
- **Vectors:** Lakehouse vectord (HNSW search, RAG)
|
||||||
|
- **AI:** Lakehouse aibridge → Ollama
|
||||||
|
- **New:** Upload portal (standalone HTML/JS — NOT Dioxus WASM)
|
||||||
|
|
||||||
|
## Why HTML/JS for the Upload Portal
|
||||||
|
|
||||||
|
- No 39MB WASM payload
|
||||||
|
- Chunked file upload (XMLHttpRequest with progress)
|
||||||
|
- Works on any browser
|
||||||
|
- Fast to iterate
|
||||||
|
- The Dioxus UI can coexist for the SQL/Ask features
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phases
|
||||||
|
|
||||||
|
### Phase 1: Upload Portal
|
||||||
|
- Chunked file upload (handles 100MB+ PDFs)
|
||||||
|
- Upload progress bar per file
|
||||||
|
- Batch upload (drag & drop multiple)
|
||||||
|
- File list with status (uploading, processing, done, failed)
|
||||||
|
- Served as static HTML from the gateway
|
||||||
|
|
||||||
|
### Phase 2: Extraction Pipeline
|
||||||
|
- ChatGPT PDF parser (conversation turns, titles, roles)
|
||||||
|
- Claude PDF parser (similar structure)
|
||||||
|
- Generic PDF fallback (page-based)
|
||||||
|
- Extraction preview: see what was extracted before committing
|
||||||
|
- Auto-embed extracted turns
|
||||||
|
|
||||||
|
### Phase 3: Knowledge Browser
|
||||||
|
- Search across all conversations by meaning
|
||||||
|
- Filter by date, topic, source
|
||||||
|
- View individual conversations with turns
|
||||||
|
- Highlight matching content in search results
|
||||||
|
|
||||||
|
### Phase 4: Comparison Tools
|
||||||
|
- Side-by-side conversation comparison
|
||||||
|
- Topic tracking across conversations over time
|
||||||
|
- "Best approach" extraction from successful conversations
|
||||||
|
- Playbook builder: curate successful patterns
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Data Model
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Conversations (one per PDF/chat export)
|
||||||
|
conversations:
|
||||||
|
conversation_id TEXT PRIMARY KEY
|
||||||
|
title TEXT
|
||||||
|
source TEXT -- "chatgpt", "claude", "other"
|
||||||
|
source_file TEXT
|
||||||
|
date TEXT
|
||||||
|
turn_count INTEGER
|
||||||
|
total_chars INTEGER
|
||||||
|
topics TEXT -- pipe-separated
|
||||||
|
summary TEXT -- AI-generated
|
||||||
|
|
||||||
|
-- Individual turns
|
||||||
|
turns:
|
||||||
|
turn_id TEXT PRIMARY KEY
|
||||||
|
conversation_id TEXT
|
||||||
|
turn_number INTEGER
|
||||||
|
role TEXT -- "user", "assistant"
|
||||||
|
content TEXT
|
||||||
|
char_count INTEGER
|
||||||
|
has_code BOOLEAN
|
||||||
|
page_number INTEGER
|
||||||
|
|
||||||
|
-- Vector embeddings (via vectord)
|
||||||
|
-- Stored in vectors/observer_kb.parquet
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Upload Requirements
|
||||||
|
|
||||||
|
- Max file size: 500MB per PDF
|
||||||
|
- Chunked upload: 5MB chunks with resume on failure
|
||||||
|
- Batch: up to 50 files at once
|
||||||
|
- Progress: per-file percentage + overall
|
||||||
|
- Dedup: skip already-ingested files (content hash)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Non-Goals (for now)
|
||||||
|
|
||||||
|
- Live chat ingestion (only PDF exports)
|
||||||
|
- Editing conversations
|
||||||
|
- Multi-user access control
|
||||||
|
- Real-time collaboration
|
||||||
@@ -0,0 +1,178 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
|
<title>Observer KB</title>
|
||||||
|
<style>
|
||||||
|
:root{--bg:#0a0a0f;--surface:#12121a;--border:#2a2a3a;--text:#e0e0e8;--dim:#888898;--accent:#6c5ce7;--success:#00d2a0;--error:#ff6b6b;--mono:'JetBrains Mono','Fira Code',monospace}
|
||||||
|
*{margin:0;padding:0;box-sizing:border-box}
|
||||||
|
body{font-family:var(--mono);background:var(--bg);color:var(--text);font-size:13px;min-height:100vh}
|
||||||
|
.app{max-width:1100px;margin:0 auto;padding:24px}
|
||||||
|
header{display:flex;align-items:center;justify-content:space-between;padding:16px 0;border-bottom:1px solid var(--border);margin-bottom:24px}
|
||||||
|
header h1{font-size:20px;color:var(--accent)}
|
||||||
|
.stats{font-size:11px;color:var(--dim)}
|
||||||
|
.tabs{display:flex;gap:2px;margin-bottom:24px}
|
||||||
|
.tab{background:none;border:none;color:var(--dim);font-family:var(--mono);font-size:13px;padding:10px 20px;cursor:pointer;border-bottom:2px solid transparent}
|
||||||
|
.tab:hover{color:var(--text)}.tab.active{color:var(--accent);border-bottom-color:var(--accent)}
|
||||||
|
.upload-zone{border:2px dashed var(--border);border-radius:12px;padding:48px 24px;text-align:center;cursor:pointer;transition:all .2s;margin-bottom:24px}
|
||||||
|
.upload-zone:hover,.upload-zone.dragover{border-color:var(--accent);background:rgba(108,92,231,.05)}
|
||||||
|
.upload-zone h3{font-size:16px;margin-bottom:8px}
|
||||||
|
.upload-zone p{color:var(--dim);font-size:12px}
|
||||||
|
.file-list{display:flex;flex-direction:column;gap:8px}
|
||||||
|
.file-item{background:var(--surface);border:1px solid var(--border);border-radius:8px;padding:14px 18px;display:grid;grid-template-columns:1fr 100px 120px;align-items:center;gap:12px}
|
||||||
|
.file-name{font-weight:600;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}
|
||||||
|
.file-size{color:var(--dim);font-size:11px;text-align:right}
|
||||||
|
.file-status{font-size:11px;text-align:right}
|
||||||
|
.uploading{color:var(--accent)}.done{color:var(--success)}.failed{color:var(--error)}
|
||||||
|
.progress-bar{height:3px;background:var(--border);border-radius:2px;margin-top:6px;grid-column:1/-1}
|
||||||
|
.progress-fill{height:100%;background:var(--accent);border-radius:2px;transition:width .3s}
|
||||||
|
.search-bar{display:flex;gap:8px;margin-bottom:20px}
|
||||||
|
.search-bar input{flex:1;background:var(--surface);border:2px solid var(--border);color:var(--text);font-family:var(--mono);font-size:14px;padding:12px 16px;border-radius:8px;outline:none}
|
||||||
|
.search-bar input:focus{border-color:var(--accent)}
|
||||||
|
.btn{background:var(--accent);color:white;border:none;padding:10px 20px;border-radius:6px;font-family:var(--mono);font-size:13px;cursor:pointer;font-weight:600}
|
||||||
|
.btn:hover{opacity:.9}
|
||||||
|
.ds-item{background:var(--surface);border:1px solid var(--border);border-radius:8px;padding:16px;margin-bottom:8px;cursor:pointer;transition:all .15s}
|
||||||
|
.ds-item:hover{border-color:var(--accent)}
|
||||||
|
.ds-title{font-weight:600;font-size:14px;margin-bottom:4px}
|
||||||
|
.ds-meta{font-size:11px;color:var(--dim)}
|
||||||
|
.hidden{display:none}
|
||||||
|
.empty{color:var(--dim);text-align:center;padding:32px}
|
||||||
|
.result-item{background:var(--surface);border:1px solid var(--border);border-radius:8px;padding:14px;margin-bottom:8px;border-left:3px solid var(--accent)}
|
||||||
|
.result-score{font-size:11px;color:var(--accent);margin-bottom:4px}
|
||||||
|
.result-text{font-size:12px;line-height:1.6;color:var(--dim)}
|
||||||
|
#output{background:var(--surface);border:1px solid var(--border);border-radius:8px;padding:16px;margin-top:16px;font-size:12px;line-height:1.6;max-height:500px;overflow-y:auto;white-space:pre-wrap}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="app">
|
||||||
|
<header><h1>OBSERVER KB</h1><div class="stats" id="stats">loading...</div></header>
|
||||||
|
<div class="tabs">
|
||||||
|
<button class="tab active" id="t-upload" onclick="showTab('upload')">Upload</button>
|
||||||
|
<button class="tab" id="t-browse" onclick="showTab('browse')">Browse</button>
|
||||||
|
<button class="tab" id="t-search" onclick="showTab('search')">Search</button>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div id="tab-upload">
|
||||||
|
<div class="upload-zone" id="dropZone">
|
||||||
|
<h3>Drop PDF files here</h3>
|
||||||
|
<p>ChatGPT exports, Claude logs, any PDF — or click to browse</p>
|
||||||
|
<input type="file" id="fileInput" multiple accept=".pdf,.txt,.csv,.json" style="display:none">
|
||||||
|
</div>
|
||||||
|
<div class="file-list" id="fileList"></div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div id="tab-browse" class="hidden">
|
||||||
|
<div id="dsList"></div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div id="tab-search" class="hidden">
|
||||||
|
<div class="search-bar">
|
||||||
|
<input type="text" id="searchInput" placeholder="Search across all knowledge..." />
|
||||||
|
<button class="btn" id="searchBtn">Search</button>
|
||||||
|
</div>
|
||||||
|
<div id="searchResults"></div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<script>
|
||||||
|
const API=`http://${location.hostname}:3100`;
|
||||||
|
const $=id=>document.getElementById(id);
|
||||||
|
|
||||||
|
function showTab(n){
|
||||||
|
document.querySelectorAll('[id^="tab-"]').forEach(t=>t.classList.add('hidden'));
|
||||||
|
$('tab-'+n).classList.remove('hidden');
|
||||||
|
document.querySelectorAll('.tab').forEach(t=>t.classList.remove('active'));
|
||||||
|
$('t-'+n).classList.add('active');
|
||||||
|
if(n==='browse')loadDatasets();
|
||||||
|
}
|
||||||
|
|
||||||
|
$('dropZone').onclick=()=>$('fileInput').click();
|
||||||
|
$('dropZone').ondragover=e=>{e.preventDefault();$('dropZone').classList.add('dragover')};
|
||||||
|
$('dropZone').ondragleave=()=>$('dropZone').classList.remove('dragover');
|
||||||
|
$('dropZone').ondrop=e=>{e.preventDefault();$('dropZone').classList.remove('dragover');[...e.dataTransfer.files].forEach(uploadFile)};
|
||||||
|
$('fileInput').onchange=e=>[...e.target.files].forEach(uploadFile);
|
||||||
|
$('searchBtn').onclick=doSearch;
|
||||||
|
$('searchInput').onkeydown=e=>{if(e.key==='Enter')doSearch()};
|
||||||
|
|
||||||
|
function uploadFile(file){
|
||||||
|
const id='f'+Date.now()+Math.random().toString(36).substr(2,5);
|
||||||
|
const item=document.createElement('div');
|
||||||
|
item.className='file-item';
|
||||||
|
item.id=id;
|
||||||
|
const nameDiv=document.createElement('div');nameDiv.className='file-name';nameDiv.textContent=file.name;
|
||||||
|
const sizeDiv=document.createElement('div');sizeDiv.className='file-size';sizeDiv.textContent=(file.size/1024/1024).toFixed(1)+' MB';
|
||||||
|
const statusDiv=document.createElement('div');statusDiv.className='file-status uploading';statusDiv.textContent='uploading...';
|
||||||
|
const progBar=document.createElement('div');progBar.className='progress-bar';
|
||||||
|
const progFill=document.createElement('div');progFill.className='progress-fill';progFill.style.width='0%';
|
||||||
|
progBar.appendChild(progFill);
|
||||||
|
item.append(nameDiv,sizeDiv,statusDiv,progBar);
|
||||||
|
$('fileList').prepend(item);
|
||||||
|
|
||||||
|
const fd=new FormData();fd.append('file',file);
|
||||||
|
const xhr=new XMLHttpRequest();
|
||||||
|
xhr.open('POST',API+'/ingest/file');
|
||||||
|
xhr.upload.onprogress=e=>{if(e.lengthComputable){const p=Math.round(e.loaded/e.total*100);progFill.style.width=p+'%';statusDiv.textContent=p+'%'}};
|
||||||
|
xhr.onload=()=>{
|
||||||
|
if(xhr.status<300){statusDiv.className='file-status done';statusDiv.textContent='done';progFill.style.background='var(--success)';loadStats()}
|
||||||
|
else{statusDiv.className='file-status failed';statusDiv.textContent='failed';progFill.style.background='var(--error)'}
|
||||||
|
};
|
||||||
|
xhr.onerror=()=>{statusDiv.className='file-status failed';statusDiv.textContent='error'};
|
||||||
|
xhr.send(fd);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loadStats(){
|
||||||
|
try{const r=await(await fetch(API+'/catalog/datasets')).json();$('stats').textContent=r.length+' datasets'}
|
||||||
|
catch{$('stats').textContent='offline'}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loadDatasets(){
|
||||||
|
try{
|
||||||
|
const ds=await(await fetch(API+'/catalog/datasets')).json();
|
||||||
|
const el=$('dsList');el.textContent='';
|
||||||
|
if(!ds.length){el.textContent='No data yet';return}
|
||||||
|
ds.forEach(d=>{
|
||||||
|
const item=document.createElement('div');item.className='ds-item';
|
||||||
|
const title=document.createElement('div');title.className='ds-title';title.textContent=d.name;
|
||||||
|
const meta=document.createElement('div');meta.className='ds-meta';
|
||||||
|
meta.textContent=`${d.row_count||'?'} rows · ${d.objects.length} file(s) · ${(d.sensitivity||'unclassified')}`;
|
||||||
|
item.append(title,meta);
|
||||||
|
item.onclick=()=>previewDS(d.name);
|
||||||
|
el.appendChild(item);
|
||||||
|
});
|
||||||
|
}catch(e){$('dsList').textContent='Error: '+e}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function previewDS(name){
|
||||||
|
try{
|
||||||
|
const r=await(await fetch(API+'/query/sql',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({sql:`SELECT * FROM ${name} LIMIT 20`})})).json();
|
||||||
|
const el=$('dsList');el.textContent='';
|
||||||
|
const btn=document.createElement('button');btn.className='btn';btn.textContent='Back';btn.onclick=loadDatasets;
|
||||||
|
const pre=document.createElement('div');pre.id='output';
|
||||||
|
pre.textContent=JSON.stringify(r.rows,null,2);
|
||||||
|
el.append(btn,pre);
|
||||||
|
}catch(e){alert(e)}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function doSearch(){
|
||||||
|
const q=$('searchInput').value;if(!q)return;
|
||||||
|
const el=$('searchResults');el.textContent='searching...';
|
||||||
|
try{
|
||||||
|
const r=await fetch(API+'/vectors/hnsw/search',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({index_name:'observer_kb',query:q,top_k:10})});
|
||||||
|
if(r.ok){
|
||||||
|
const d=await r.json();el.textContent='';
|
||||||
|
d.results.forEach(hit=>{
|
||||||
|
const item=document.createElement('div');item.className='result-item';
|
||||||
|
const score=document.createElement('div');score.className='result-score';score.textContent=`[${hit.score.toFixed(3)}] ${hit.source} / ${hit.doc_id}`;
|
||||||
|
const text=document.createElement('div');text.className='result-text';text.textContent=hit.chunk_text.substring(0,300);
|
||||||
|
item.append(score,text);el.appendChild(item);
|
||||||
|
});
|
||||||
|
if(!d.results.length)el.textContent='No results';
|
||||||
|
}else{el.textContent='Vector index not built yet — upload PDFs first'}
|
||||||
|
}catch(e){el.textContent='Error: '+e}
|
||||||
|
}
|
||||||
|
|
||||||
|
loadStats();
|
||||||
|
</script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,235 @@
|
|||||||
|
/// ChatGPT PDF conversation parser.
|
||||||
|
/// Extracts structured conversation turns from ChatGPT export PDFs.
|
||||||
|
/// Handles: user/assistant turns, timestamps, conversation titles.
|
||||||
|
|
||||||
|
use arrow::array::{ArrayRef, Int32Array, RecordBatch, StringArray};
|
||||||
|
use arrow::datatypes::{DataType, Field, Schema};
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
/// A single conversation turn.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct ConversationTurn {
|
||||||
|
pub source_file: String,
|
||||||
|
pub conversation_title: String,
|
||||||
|
pub turn_number: i32,
|
||||||
|
pub role: String, // "user" or "assistant"
|
||||||
|
pub content: String,
|
||||||
|
pub page_number: i32,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Parse a ChatGPT PDF into structured conversation turns.
|
||||||
|
pub fn parse_chatgpt_pdf(content: &[u8], filename: &str) -> Result<(Arc<Schema>, Vec<RecordBatch>), String> {
|
||||||
|
let doc = lopdf::Document::load_mem(content)
|
||||||
|
.map_err(|e| format!("PDF load error: {e}"))?;
|
||||||
|
|
||||||
|
// Extract all text
|
||||||
|
let pages = doc.get_pages();
|
||||||
|
let mut all_text = String::new();
|
||||||
|
let mut page_texts: Vec<(i32, String)> = Vec::new();
|
||||||
|
|
||||||
|
for (&page_num, _) in pages.iter() {
|
||||||
|
let text = doc.extract_text(&[page_num]).unwrap_or_default();
|
||||||
|
let text = text.trim().to_string();
|
||||||
|
if !text.is_empty() {
|
||||||
|
page_texts.push((page_num as i32, text.clone()));
|
||||||
|
all_text.push_str(&text);
|
||||||
|
all_text.push('\n');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if all_text.trim().is_empty() {
|
||||||
|
return Err("PDF contains no extractable text".into());
|
||||||
|
}
|
||||||
|
|
||||||
|
// Try to parse as ChatGPT conversation format
|
||||||
|
let turns = extract_turns(&all_text, filename, &page_texts);
|
||||||
|
|
||||||
|
if turns.is_empty() {
|
||||||
|
// Fall back to page-based extraction if we can't detect conversation structure
|
||||||
|
return parse_as_raw_pages(&page_texts, filename);
|
||||||
|
}
|
||||||
|
|
||||||
|
tracing::info!("parsed ChatGPT PDF '{}': {} turns", filename, turns.len());
|
||||||
|
turns_to_batch(&turns)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Extract conversation turns from ChatGPT PDF text.
|
||||||
|
/// ChatGPT PDFs typically have patterns like:
|
||||||
|
/// - "You said:" / "ChatGPT said:" (newer format)
|
||||||
|
/// - "User:" / "Assistant:"
|
||||||
|
/// - Or just alternating blocks separated by formatting
|
||||||
|
fn extract_turns(text: &str, filename: &str, page_texts: &[(i32, String)]) -> Vec<ConversationTurn> {
|
||||||
|
let mut turns = Vec::new();
|
||||||
|
let mut turn_num = 0;
|
||||||
|
|
||||||
|
// Try to extract conversation title from first line
|
||||||
|
let title = text.lines().next()
|
||||||
|
.map(|l| l.trim().to_string())
|
||||||
|
.filter(|l| l.len() < 200 && !l.is_empty())
|
||||||
|
.unwrap_or_else(|| filename.replace(".pdf", ""));
|
||||||
|
|
||||||
|
// Strategy 1: Look for "You said:" / "ChatGPT said:" markers (common format)
|
||||||
|
let you_markers = ["You said:", "You:", "User:", "Human:"];
|
||||||
|
let ai_markers = ["ChatGPT said:", "ChatGPT:", "Assistant:", "AI:"];
|
||||||
|
|
||||||
|
let has_markers = you_markers.iter().any(|m| text.contains(m))
|
||||||
|
&& ai_markers.iter().any(|m| text.contains(m));
|
||||||
|
|
||||||
|
if has_markers {
|
||||||
|
let lines: Vec<&str> = text.lines().collect();
|
||||||
|
let mut current_role = String::new();
|
||||||
|
let mut current_content = String::new();
|
||||||
|
let mut current_page = 1i32;
|
||||||
|
|
||||||
|
for line in &lines {
|
||||||
|
let trimmed = line.trim();
|
||||||
|
|
||||||
|
// Check for role markers
|
||||||
|
let mut found_marker = false;
|
||||||
|
for marker in &you_markers {
|
||||||
|
if trimmed.starts_with(marker) || trimmed == *marker {
|
||||||
|
// Save previous turn
|
||||||
|
if !current_content.trim().is_empty() && !current_role.is_empty() {
|
||||||
|
turn_num += 1;
|
||||||
|
turns.push(ConversationTurn {
|
||||||
|
source_file: filename.to_string(),
|
||||||
|
conversation_title: title.clone(),
|
||||||
|
turn_number: turn_num,
|
||||||
|
role: current_role.clone(),
|
||||||
|
content: current_content.trim().to_string(),
|
||||||
|
page_number: current_page,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
current_role = "user".to_string();
|
||||||
|
current_content = trimmed.strip_prefix(marker).unwrap_or("").trim().to_string();
|
||||||
|
current_content.push('\n');
|
||||||
|
found_marker = true;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !found_marker {
|
||||||
|
for marker in &ai_markers {
|
||||||
|
if trimmed.starts_with(marker) || trimmed == *marker {
|
||||||
|
if !current_content.trim().is_empty() && !current_role.is_empty() {
|
||||||
|
turn_num += 1;
|
||||||
|
turns.push(ConversationTurn {
|
||||||
|
source_file: filename.to_string(),
|
||||||
|
conversation_title: title.clone(),
|
||||||
|
turn_number: turn_num,
|
||||||
|
role: current_role.clone(),
|
||||||
|
content: current_content.trim().to_string(),
|
||||||
|
page_number: current_page,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
current_role = "assistant".to_string();
|
||||||
|
current_content = trimmed.strip_prefix(marker).unwrap_or("").trim().to_string();
|
||||||
|
current_content.push('\n');
|
||||||
|
found_marker = true;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if !found_marker && !current_role.is_empty() {
|
||||||
|
current_content.push_str(trimmed);
|
||||||
|
current_content.push('\n');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Save last turn
|
||||||
|
if !current_content.trim().is_empty() && !current_role.is_empty() {
|
||||||
|
turn_num += 1;
|
||||||
|
turns.push(ConversationTurn {
|
||||||
|
source_file: filename.to_string(),
|
||||||
|
conversation_title: title.clone(),
|
||||||
|
turn_number: turn_num,
|
||||||
|
role: current_role,
|
||||||
|
content: current_content.trim().to_string(),
|
||||||
|
page_number: current_page,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
return turns;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Strategy 2: Split by paragraph blocks, alternate user/assistant
|
||||||
|
// This is a heuristic fallback for PDFs without clear markers
|
||||||
|
let paragraphs: Vec<&str> = text.split("\n\n")
|
||||||
|
.map(|p| p.trim())
|
||||||
|
.filter(|p| p.len() > 20) // skip short fragments
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
if paragraphs.len() >= 2 {
|
||||||
|
for (i, para) in paragraphs.iter().enumerate() {
|
||||||
|
turn_num += 1;
|
||||||
|
turns.push(ConversationTurn {
|
||||||
|
source_file: filename.to_string(),
|
||||||
|
conversation_title: title.clone(),
|
||||||
|
turn_number: turn_num,
|
||||||
|
role: if i % 2 == 0 { "user" } else { "assistant" }.to_string(),
|
||||||
|
content: para.to_string(),
|
||||||
|
page_number: 1,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
turns
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Fallback: parse as raw pages with full text.
|
||||||
|
fn parse_as_raw_pages(page_texts: &[(i32, String)], filename: &str) -> Result<(Arc<Schema>, Vec<RecordBatch>), String> {
|
||||||
|
let schema = Arc::new(Schema::new(vec![
|
||||||
|
Field::new("source_file", DataType::Utf8, false),
|
||||||
|
Field::new("conversation_title", DataType::Utf8, false),
|
||||||
|
Field::new("turn_number", DataType::Int32, false),
|
||||||
|
Field::new("role", DataType::Utf8, false),
|
||||||
|
Field::new("content", DataType::Utf8, false),
|
||||||
|
Field::new("page_number", DataType::Int32, false),
|
||||||
|
]));
|
||||||
|
|
||||||
|
let sources: Vec<&str> = page_texts.iter().map(|_| filename).collect();
|
||||||
|
let titles: Vec<String> = page_texts.iter().map(|_| filename.replace(".pdf", "")).collect();
|
||||||
|
let turn_nums: Vec<i32> = (1..=page_texts.len() as i32).collect();
|
||||||
|
let roles: Vec<&str> = page_texts.iter().map(|_| "page").collect();
|
||||||
|
let contents: Vec<&str> = page_texts.iter().map(|(_, t)| t.as_str()).collect();
|
||||||
|
let pages: Vec<i32> = page_texts.iter().map(|(p, _)| *p).collect();
|
||||||
|
|
||||||
|
let arrays: Vec<ArrayRef> = vec![
|
||||||
|
Arc::new(StringArray::from(sources)),
|
||||||
|
Arc::new(StringArray::from(titles.iter().map(|s| s.as_str()).collect::<Vec<_>>())),
|
||||||
|
Arc::new(Int32Array::from(turn_nums)),
|
||||||
|
Arc::new(StringArray::from(roles)),
|
||||||
|
Arc::new(StringArray::from(contents)),
|
||||||
|
Arc::new(Int32Array::from(pages)),
|
||||||
|
];
|
||||||
|
|
||||||
|
let batch = RecordBatch::try_new(schema.clone(), arrays)
|
||||||
|
.map_err(|e| format!("RecordBatch error: {e}"))?;
|
||||||
|
|
||||||
|
Ok((schema, vec![batch]))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn turns_to_batch(turns: &[ConversationTurn]) -> Result<(Arc<Schema>, Vec<RecordBatch>), String> {
|
||||||
|
let schema = Arc::new(Schema::new(vec![
|
||||||
|
Field::new("source_file", DataType::Utf8, false),
|
||||||
|
Field::new("conversation_title", DataType::Utf8, false),
|
||||||
|
Field::new("turn_number", DataType::Int32, false),
|
||||||
|
Field::new("role", DataType::Utf8, false),
|
||||||
|
Field::new("content", DataType::Utf8, false),
|
||||||
|
Field::new("page_number", DataType::Int32, false),
|
||||||
|
]));
|
||||||
|
|
||||||
|
let arrays: Vec<ArrayRef> = vec![
|
||||||
|
Arc::new(StringArray::from(turns.iter().map(|t| t.source_file.as_str()).collect::<Vec<_>>())),
|
||||||
|
Arc::new(StringArray::from(turns.iter().map(|t| t.conversation_title.as_str()).collect::<Vec<_>>())),
|
||||||
|
Arc::new(Int32Array::from(turns.iter().map(|t| t.turn_number).collect::<Vec<_>>())),
|
||||||
|
Arc::new(StringArray::from(turns.iter().map(|t| t.role.as_str()).collect::<Vec<_>>())),
|
||||||
|
Arc::new(StringArray::from(turns.iter().map(|t| t.content.as_str()).collect::<Vec<_>>())),
|
||||||
|
Arc::new(Int32Array::from(turns.iter().map(|t| t.page_number).collect::<Vec<_>>())),
|
||||||
|
];
|
||||||
|
|
||||||
|
let batch = RecordBatch::try_new(schema.clone(), arrays)
|
||||||
|
.map_err(|e| format!("RecordBatch error: {e}"))?;
|
||||||
|
|
||||||
|
Ok((schema, vec![batch]))
|
||||||
|
}
|
||||||
@@ -1,3 +1,4 @@
|
|||||||
|
pub mod chatgpt_parser;
|
||||||
pub mod db_ingest;
|
pub mod db_ingest;
|
||||||
pub mod detect;
|
pub mod detect;
|
||||||
pub mod csv_ingest;
|
pub mod csv_ingest;
|
||||||
|
|||||||
@@ -43,7 +43,13 @@ pub async fn ingest_file(
|
|||||||
let (schema, batches) = match file_type {
|
let (schema, batches) = match file_type {
|
||||||
FileType::Csv => csv_ingest::parse_csv(content)?,
|
FileType::Csv => csv_ingest::parse_csv(content)?,
|
||||||
FileType::Json | FileType::NdJson => json_ingest::parse_json(content)?,
|
FileType::Json | FileType::NdJson => json_ingest::parse_json(content)?,
|
||||||
FileType::Pdf => pdf_ingest::parse_pdf(content, filename)?,
|
FileType::Pdf => {
|
||||||
|
// Try ChatGPT parser first, fall back to generic PDF
|
||||||
|
match crate::chatgpt_parser::parse_chatgpt_pdf(content, filename) {
|
||||||
|
Ok(result) => result,
|
||||||
|
Err(_) => pdf_ingest::parse_pdf(content, filename)?,
|
||||||
|
}
|
||||||
|
}
|
||||||
FileType::Text => parse_text_file(content, filename)?,
|
FileType::Text => parse_text_file(content, filename)?,
|
||||||
FileType::Unknown => return Err(format!("unknown file type for '{filename}'")),
|
FileType::Unknown => return Err(format!("unknown file type for '{filename}'")),
|
||||||
};
|
};
|
||||||
|
|||||||
Reference in New Issue
Block a user