Update PRD + memory: Observer KB needs devop.live LLM Team mode integration
Observer KB PRD v2: - Multi-pass pipeline using devop.live's 20 analysis modes - extract → validate → hallucination-check → consensus → organize → render - Q&A chain extraction (questions as anchors) - Curated document output (hand-it-to-someone quality) - Integration architecture with LLM Team API on :5000 Memory updated with: - Two-project structure (Lakehouse core + Observer KB) - What's built vs what's next - Critical: devop.live modes ARE the multi-pass pipeline - Next session: integrate Observer KB with LLM Team modes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
+145
-121
@@ -1,6 +1,6 @@
|
|||||||
# PRD: Observer Knowledge Base
|
# PRD: Observer Knowledge Base
|
||||||
|
|
||||||
**Status:** Active
|
**Status:** Active — extraction pipeline proven, needs multi-pass processing via LLM Team modes
|
||||||
**Created:** 2026-03-28
|
**Created:** 2026-03-28
|
||||||
**Owner:** J
|
**Owner:** J
|
||||||
**Branch:** `observer-kb`
|
**Branch:** `observer-kb`
|
||||||
@@ -10,150 +10,174 @@
|
|||||||
|
|
||||||
## Problem
|
## Problem
|
||||||
|
|
||||||
AI conversations (ChatGPT, Claude, etc.) generate massive amounts of knowledge — decisions, research, code, strategies, troubleshooting — that disappear into PDF exports and chat logs. There's no way to:
|
AI conversations generate massive amounts of knowledge that's lost to context windows, hallucination, and scope drift. Current approach (single-pass extraction into categories) produces disorganized, hard-to-validate output. Reading raw pages is not useful — the system needs to process, clean, structure, and curate knowledge like an editorial team.
|
||||||
- Search across hundreds of conversations by meaning
|
|
||||||
- Find "what did I discuss about X six months ago?"
|
|
||||||
- Compare approaches across different conversations
|
|
||||||
- Build a reusable knowledge base from past AI interactions
|
|
||||||
- Track which ideas worked and which didn't
|
|
||||||
|
|
||||||
## Solution
|
## Vision
|
||||||
|
|
||||||
An upload portal + knowledge extraction pipeline built on the Lakehouse infrastructure. Upload ChatGPT/Claude PDFs → extract conversations → chunk and embed → searchable knowledge base with comparison tools.
|
A publishing pipeline that takes raw AI conversation exports and produces well-curated, validated knowledge documents. The output should be something you hand to a team — organized, filtered, and clearly presented. Questions anchor the content, with branching connections showing what was asked, what was answered, and how reasoning was directed.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
|
### Multi-Pass Processing via LLM Team Modes
|
||||||
|
|
||||||
|
The key insight: devop.live already has 20 analysis modes that process content through structured workflows. The Observer KB should route uploaded data through these modes automatically.
|
||||||
|
|
||||||
```
|
```
|
||||||
Upload Portal (web UI)
|
Raw PDF Upload
|
||||||
│
|
│
|
||||||
├── Upload large PDFs (chunked upload for big files)
|
▼ Mode: EXTRACT
|
||||||
├── Progress tracking per file
|
Structured facts, entities, relationships
|
||||||
└── Visual extraction preview
|
|
||||||
│
|
│
|
||||||
▼
|
▼ Mode: VALIDATOR
|
||||||
Extraction Pipeline (uses Lakehouse ingestd)
|
Fact-check claims, score 1-10 confidence
|
||||||
│
|
│
|
||||||
├── PDF text extraction (lopdf)
|
▼ Mode: HALLUCINATION
|
||||||
├── Conversation turn parsing (user/assistant)
|
Verify each claim, flag likely hallucinations
|
||||||
├── Topic detection
|
|
||||||
├── Metadata extraction (dates, titles, participants)
|
|
||||||
└── Quality scoring (how useful was this conversation?)
|
|
||||||
│
|
│
|
||||||
▼
|
▼ Mode: CONSENSUS
|
||||||
Lakehouse Storage
|
Multiple passes iterate until extraction converges
|
||||||
│
|
│
|
||||||
├── conversations table (id, title, source, date, turns, topic)
|
▼ Mode: REDTEAM
|
||||||
├── turns table (conversation_id, turn_num, role, content)
|
Find flaws in knowledge, patch them
|
||||||
├── topics table (topic, conversation_ids, summary)
|
|
||||||
└── vectors/ (embedded turns for semantic search)
|
|
||||||
│
|
│
|
||||||
▼
|
▼ Mode: PIPELINE (custom)
|
||||||
Knowledge Interface
|
Clean → Q&A structure → topic grouping → value ranking
|
||||||
│
|
│
|
||||||
├── Search: "what did I discuss about Kubernetes?"
|
▼ RENDER
|
||||||
├── Compare: conversation A vs B on same topic
|
Well-curated document: questions as anchors, branching connections,
|
||||||
├── Timeline: when did I first explore this idea?
|
what was asked → answered → directed
|
||||||
├── Extract: pull key decisions/code/strategies
|
|
||||||
└── Playbook: which approaches worked?
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Integration with devop.live (LLM Team)
|
||||||
|
|
||||||
|
The Observer KB calls the LLM Team API to run modes against extracted content:
|
||||||
|
|
||||||
|
| Mode | Purpose in KB |
|
||||||
|
|---|---|
|
||||||
|
| **extract** | Structured facts, entities, relationships from raw text |
|
||||||
|
| **validator** | Fact-check extracted claims, confidence scores |
|
||||||
|
| **hallucination** | Verify each factual claim independently |
|
||||||
|
| **drift** | Process N times, find consistent vs uncertain content |
|
||||||
|
| **redteam** | Find flaws in extracted knowledge, improve |
|
||||||
|
| **consensus** | Iterate until multiple passes agree on correct extraction |
|
||||||
|
| **pipeline** | Chain: clean → extract → validate → organize → render |
|
||||||
|
| **research** | Generate follow-up questions, research in parallel |
|
||||||
|
| **blindassembly** | Split content into parts, process independently, reassemble |
|
||||||
|
|
||||||
|
### Data Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
Upload Portal (Observer KB :3400)
|
||||||
|
│
|
||||||
|
├── PDF → Lakehouse ingestd (text extraction, Parquet storage)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Processing Pipeline
|
||||||
|
│
|
||||||
|
├── Call devop.live API (:5000) with each mode
|
||||||
|
├── Each mode processes content through local Ollama models
|
||||||
|
├── Results stored back in Lakehouse (Parquet + catalog)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Knowledge Store
|
||||||
|
│
|
||||||
|
├── knowledge_facts: verified claims with confidence scores
|
||||||
|
├── knowledge_qa: question → answer pairs with context
|
||||||
|
├── knowledge_topics: auto-generated categories
|
||||||
|
├── knowledge_connections: how topics relate across conversations
|
||||||
|
├── knowledge_contradictions: flagged conflicts between conversations
|
||||||
|
├── knowledge_validated: claims that passed hallucination check
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Rendering
|
||||||
|
│
|
||||||
|
├── Structured knowledge view (web UI)
|
||||||
|
├── Exportable curated document (PDF/markdown)
|
||||||
|
└── Question-anchored navigation (Q → A → follow-up → related)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Processing Pipeline Detail
|
||||||
|
|
||||||
|
### Pass 1: Clean + Structure
|
||||||
|
- Strip PDF artifacts (headers, footers, page numbers)
|
||||||
|
- Identify conversation turns (user/AI)
|
||||||
|
- Fix encoding issues (ligatures, special chars)
|
||||||
|
- Normalize formatting
|
||||||
|
|
||||||
|
### Pass 2: Q&A Extraction
|
||||||
|
- Identify questions asked by user
|
||||||
|
- Map each question to its answer
|
||||||
|
- Track follow-up questions and how they branch
|
||||||
|
- Build question→answer→follow-up chains
|
||||||
|
|
||||||
|
### Pass 3: Fact Extraction + Validation
|
||||||
|
- Run EXTRACT mode: pull structured facts, entities, relationships
|
||||||
|
- Run VALIDATOR mode: score each fact 1-10 confidence
|
||||||
|
- Run HALLUCINATION mode: verify claims independently
|
||||||
|
- Flag contradictions between conversations
|
||||||
|
|
||||||
|
### Pass 4: Organize + Curate
|
||||||
|
- Run CONSENSUS mode: multiple passes converge on correct categorization
|
||||||
|
- Auto-generate topic categories from content (not predefined)
|
||||||
|
- Rank content by value (decisions > insights > discussion > noise)
|
||||||
|
- Connect related content across conversations
|
||||||
|
|
||||||
|
### Pass 5: Render
|
||||||
|
- Question-anchored view: each question is a node, answers branch from it
|
||||||
|
- Topic-grouped view: knowledge organized by auto-generated categories
|
||||||
|
- Timeline view: how thinking evolved across conversations
|
||||||
|
- Export as clean markdown/PDF
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current State (as of 2026-03-28)
|
||||||
|
|
||||||
|
### What's Built
|
||||||
|
- PDF ingestion (8 PDFs, 477 pages extracted)
|
||||||
|
- Single-pass knowledge extraction (496 topics, 300 decisions, 100 insights)
|
||||||
|
- Semantic search via HNSW (27ms over 477 chunks)
|
||||||
|
- RAG (question → search → LLM answer with sources)
|
||||||
|
- 5-pass deep analysis (corpus overview, patterns, evolution, gaps, synthesis)
|
||||||
|
- Observer KB web UI (knowledge-first, not page-first)
|
||||||
|
- Upload portal with drag & drop
|
||||||
|
|
||||||
|
### What's Not Built Yet
|
||||||
|
- Integration with devop.live LLM Team modes
|
||||||
|
- Multi-pass processing pipeline (extract → validate → hallucination check)
|
||||||
|
- Q&A chain extraction (question → answer → follow-up mapping)
|
||||||
|
- Contradiction detection between conversations
|
||||||
|
- Curated document export (PDF/markdown)
|
||||||
|
- Question-anchored navigation
|
||||||
|
- Auto-processing on upload (currently manual script)
|
||||||
|
|
||||||
|
### What Needs to Happen Next Session
|
||||||
|
1. **Integrate with devop.live API** — call modes programmatically
|
||||||
|
2. **Build the multi-pass pipeline** — extract → validate → hallucinate-check → organize
|
||||||
|
3. **Q&A chain extraction** — questions as anchors, answers branching
|
||||||
|
4. **Auto-process on upload** — no manual script, pipeline runs automatically
|
||||||
|
5. **Render curated output** — readable, hand-it-to-someone quality
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Tech Stack
|
## Tech Stack
|
||||||
|
|
||||||
Uses Lakehouse core — NOT a separate codebase:
|
|
||||||
- **Storage:** Lakehouse storaged (Parquet on object storage)
|
- **Storage:** Lakehouse storaged (Parquet on object storage)
|
||||||
- **Catalog:** Lakehouse catalogd (metadata, PII, lineage)
|
- **Catalog:** Lakehouse catalogd (metadata, PII, lineage)
|
||||||
- **Query:** Lakehouse queryd (DataFusion SQL)
|
- **Query:** Lakehouse queryd (DataFusion SQL, hot cache, paged results)
|
||||||
- **Vectors:** Lakehouse vectord (HNSW search, RAG)
|
- **Vectors:** Lakehouse vectord (HNSW search, RAG)
|
||||||
- **AI:** Lakehouse aibridge → Ollama
|
- **AI Processing:** devop.live LLM Team API (20 modes, multiple models)
|
||||||
- **New:** Upload portal (standalone HTML/JS — NOT Dioxus WASM)
|
- **AI Runtime:** Ollama (local models — qwen2.5, mistral, gemma2, nomic-embed-text)
|
||||||
|
- **UI:** Pure HTML/JS on :3400
|
||||||
## Why HTML/JS for the Upload Portal
|
|
||||||
|
|
||||||
- No 39MB WASM payload
|
|
||||||
- Chunked file upload (XMLHttpRequest with progress)
|
|
||||||
- Works on any browser
|
|
||||||
- Fast to iterate
|
|
||||||
- The Dioxus UI can coexist for the SQL/Ask features
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phases
|
|
||||||
|
|
||||||
### Phase 1: Upload Portal
|
|
||||||
- Chunked file upload (handles 100MB+ PDFs)
|
|
||||||
- Upload progress bar per file
|
|
||||||
- Batch upload (drag & drop multiple)
|
|
||||||
- File list with status (uploading, processing, done, failed)
|
|
||||||
- Served as static HTML from the gateway
|
|
||||||
|
|
||||||
### Phase 2: Extraction Pipeline
|
|
||||||
- ChatGPT PDF parser (conversation turns, titles, roles)
|
|
||||||
- Claude PDF parser (similar structure)
|
|
||||||
- Generic PDF fallback (page-based)
|
|
||||||
- Extraction preview: see what was extracted before committing
|
|
||||||
- Auto-embed extracted turns
|
|
||||||
|
|
||||||
### Phase 3: Knowledge Browser
|
|
||||||
- Search across all conversations by meaning
|
|
||||||
- Filter by date, topic, source
|
|
||||||
- View individual conversations with turns
|
|
||||||
- Highlight matching content in search results
|
|
||||||
|
|
||||||
### Phase 4: Comparison Tools
|
|
||||||
- Side-by-side conversation comparison
|
|
||||||
- Topic tracking across conversations over time
|
|
||||||
- "Best approach" extraction from successful conversations
|
|
||||||
- Playbook builder: curate successful patterns
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Data Model
|
|
||||||
|
|
||||||
```sql
|
|
||||||
-- Conversations (one per PDF/chat export)
|
|
||||||
conversations:
|
|
||||||
conversation_id TEXT PRIMARY KEY
|
|
||||||
title TEXT
|
|
||||||
source TEXT -- "chatgpt", "claude", "other"
|
|
||||||
source_file TEXT
|
|
||||||
date TEXT
|
|
||||||
turn_count INTEGER
|
|
||||||
total_chars INTEGER
|
|
||||||
topics TEXT -- pipe-separated
|
|
||||||
summary TEXT -- AI-generated
|
|
||||||
|
|
||||||
-- Individual turns
|
|
||||||
turns:
|
|
||||||
turn_id TEXT PRIMARY KEY
|
|
||||||
conversation_id TEXT
|
|
||||||
turn_number INTEGER
|
|
||||||
role TEXT -- "user", "assistant"
|
|
||||||
content TEXT
|
|
||||||
char_count INTEGER
|
|
||||||
has_code BOOLEAN
|
|
||||||
page_number INTEGER
|
|
||||||
|
|
||||||
-- Vector embeddings (via vectord)
|
|
||||||
-- Stored in vectors/observer_kb.parquet
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Upload Requirements
|
|
||||||
|
|
||||||
- Max file size: 500MB per PDF
|
|
||||||
- Chunked upload: 5MB chunks with resume on failure
|
|
||||||
- Batch: up to 50 files at once
|
|
||||||
- Progress: per-file percentage + overall
|
|
||||||
- Dedup: skip already-ingested files (content hash)
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Non-Goals (for now)
|
## Non-Goals (for now)
|
||||||
|
|
||||||
- Live chat ingestion (only PDF exports)
|
- Real-time conversation ingestion (PDF exports only)
|
||||||
- Editing conversations
|
- Multi-user collaboration
|
||||||
- Multi-user access control
|
- Cloud deployment
|
||||||
- Real-time collaboration
|
- Editing extracted knowledge from UI (read-only for now)
|
||||||
|
|||||||
Reference in New Issue
Block a user