From bd68b662b79b1019fb2e2e78e1f863e0c82a11e4 Mon Sep 17 00:00:00 2001 From: root Date: Fri, 27 Mar 2026 23:35:15 -0500 Subject: [PATCH] Update PRD + memory: Observer KB needs devop.live LLM Team mode integration MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Observer KB PRD v2: - Multi-pass pipeline using devop.live's 20 analysis modes - extract → validate → hallucination-check → consensus → organize → render - Q&A chain extraction (questions as anchors) - Curated document output (hand-it-to-someone quality) - Integration architecture with LLM Team API on :5000 Memory updated with: - Two-project structure (Lakehouse core + Observer KB) - What's built vs what's next - Critical: devop.live modes ARE the multi-pass pipeline - Next session: integrate Observer KB with LLM Team modes Co-Authored-By: Claude Opus 4.6 (1M context) --- apps/observer-kb/docs/PRD.md | 266 +++++++++++++++++++---------------- 1 file changed, 145 insertions(+), 121 deletions(-) diff --git a/apps/observer-kb/docs/PRD.md b/apps/observer-kb/docs/PRD.md index 7004967..c5a9e2c 100644 --- a/apps/observer-kb/docs/PRD.md +++ b/apps/observer-kb/docs/PRD.md @@ -1,6 +1,6 @@ # PRD: Observer Knowledge Base -**Status:** Active +**Status:** Active — extraction pipeline proven, needs multi-pass processing via LLM Team modes **Created:** 2026-03-28 **Owner:** J **Branch:** `observer-kb` @@ -10,150 +10,174 @@ ## Problem -AI conversations (ChatGPT, Claude, etc.) generate massive amounts of knowledge — decisions, research, code, strategies, troubleshooting — that disappear into PDF exports and chat logs. There's no way to: -- Search across hundreds of conversations by meaning -- Find "what did I discuss about X six months ago?" -- Compare approaches across different conversations -- Build a reusable knowledge base from past AI interactions -- Track which ideas worked and which didn't +AI conversations generate massive amounts of knowledge that's lost to context windows, hallucination, and scope drift. Current approach (single-pass extraction into categories) produces disorganized, hard-to-validate output. Reading raw pages is not useful — the system needs to process, clean, structure, and curate knowledge like an editorial team. -## Solution +## Vision -An upload portal + knowledge extraction pipeline built on the Lakehouse infrastructure. Upload ChatGPT/Claude PDFs → extract conversations → chunk and embed → searchable knowledge base with comparison tools. +A publishing pipeline that takes raw AI conversation exports and produces well-curated, validated knowledge documents. The output should be something you hand to a team — organized, filtered, and clearly presented. Questions anchor the content, with branching connections showing what was asked, what was answered, and how reasoning was directed. --- ## Architecture +### Multi-Pass Processing via LLM Team Modes + +The key insight: devop.live already has 20 analysis modes that process content through structured workflows. The Observer KB should route uploaded data through these modes automatically. + ``` -Upload Portal (web UI) +Raw PDF Upload │ - ├── Upload large PDFs (chunked upload for big files) - ├── Progress tracking per file - └── Visual extraction preview + ▼ Mode: EXTRACT + Structured facts, entities, relationships │ - ▼ -Extraction Pipeline (uses Lakehouse ingestd) + ▼ Mode: VALIDATOR + Fact-check claims, score 1-10 confidence │ - ├── PDF text extraction (lopdf) - ├── Conversation turn parsing (user/assistant) - ├── Topic detection - ├── Metadata extraction (dates, titles, participants) - └── Quality scoring (how useful was this conversation?) + ▼ Mode: HALLUCINATION + Verify each claim, flag likely hallucinations │ - ▼ -Lakehouse Storage + ▼ Mode: CONSENSUS + Multiple passes iterate until extraction converges │ - ├── conversations table (id, title, source, date, turns, topic) - ├── turns table (conversation_id, turn_num, role, content) - ├── topics table (topic, conversation_ids, summary) - └── vectors/ (embedded turns for semantic search) + ▼ Mode: REDTEAM + Find flaws in knowledge, patch them │ - ▼ -Knowledge Interface + ▼ Mode: PIPELINE (custom) + Clean → Q&A structure → topic grouping → value ranking │ - ├── Search: "what did I discuss about Kubernetes?" - ├── Compare: conversation A vs B on same topic - ├── Timeline: when did I first explore this idea? - ├── Extract: pull key decisions/code/strategies - └── Playbook: which approaches worked? + ▼ RENDER + Well-curated document: questions as anchors, branching connections, + what was asked → answered → directed ``` +### Integration with devop.live (LLM Team) + +The Observer KB calls the LLM Team API to run modes against extracted content: + +| Mode | Purpose in KB | +|---|---| +| **extract** | Structured facts, entities, relationships from raw text | +| **validator** | Fact-check extracted claims, confidence scores | +| **hallucination** | Verify each factual claim independently | +| **drift** | Process N times, find consistent vs uncertain content | +| **redteam** | Find flaws in extracted knowledge, improve | +| **consensus** | Iterate until multiple passes agree on correct extraction | +| **pipeline** | Chain: clean → extract → validate → organize → render | +| **research** | Generate follow-up questions, research in parallel | +| **blindassembly** | Split content into parts, process independently, reassemble | + +### Data Flow + +``` +Upload Portal (Observer KB :3400) + │ + ├── PDF → Lakehouse ingestd (text extraction, Parquet storage) + │ + ▼ +Processing Pipeline + │ + ├── Call devop.live API (:5000) with each mode + ├── Each mode processes content through local Ollama models + ├── Results stored back in Lakehouse (Parquet + catalog) + │ + ▼ +Knowledge Store + │ + ├── knowledge_facts: verified claims with confidence scores + ├── knowledge_qa: question → answer pairs with context + ├── knowledge_topics: auto-generated categories + ├── knowledge_connections: how topics relate across conversations + ├── knowledge_contradictions: flagged conflicts between conversations + ├── knowledge_validated: claims that passed hallucination check + │ + ▼ +Rendering + │ + ├── Structured knowledge view (web UI) + ├── Exportable curated document (PDF/markdown) + └── Question-anchored navigation (Q → A → follow-up → related) +``` + +--- + +## Processing Pipeline Detail + +### Pass 1: Clean + Structure +- Strip PDF artifacts (headers, footers, page numbers) +- Identify conversation turns (user/AI) +- Fix encoding issues (ligatures, special chars) +- Normalize formatting + +### Pass 2: Q&A Extraction +- Identify questions asked by user +- Map each question to its answer +- Track follow-up questions and how they branch +- Build question→answer→follow-up chains + +### Pass 3: Fact Extraction + Validation +- Run EXTRACT mode: pull structured facts, entities, relationships +- Run VALIDATOR mode: score each fact 1-10 confidence +- Run HALLUCINATION mode: verify claims independently +- Flag contradictions between conversations + +### Pass 4: Organize + Curate +- Run CONSENSUS mode: multiple passes converge on correct categorization +- Auto-generate topic categories from content (not predefined) +- Rank content by value (decisions > insights > discussion > noise) +- Connect related content across conversations + +### Pass 5: Render +- Question-anchored view: each question is a node, answers branch from it +- Topic-grouped view: knowledge organized by auto-generated categories +- Timeline view: how thinking evolved across conversations +- Export as clean markdown/PDF + +--- + +## Current State (as of 2026-03-28) + +### What's Built +- PDF ingestion (8 PDFs, 477 pages extracted) +- Single-pass knowledge extraction (496 topics, 300 decisions, 100 insights) +- Semantic search via HNSW (27ms over 477 chunks) +- RAG (question → search → LLM answer with sources) +- 5-pass deep analysis (corpus overview, patterns, evolution, gaps, synthesis) +- Observer KB web UI (knowledge-first, not page-first) +- Upload portal with drag & drop + +### What's Not Built Yet +- Integration with devop.live LLM Team modes +- Multi-pass processing pipeline (extract → validate → hallucination check) +- Q&A chain extraction (question → answer → follow-up mapping) +- Contradiction detection between conversations +- Curated document export (PDF/markdown) +- Question-anchored navigation +- Auto-processing on upload (currently manual script) + +### What Needs to Happen Next Session +1. **Integrate with devop.live API** — call modes programmatically +2. **Build the multi-pass pipeline** — extract → validate → hallucinate-check → organize +3. **Q&A chain extraction** — questions as anchors, answers branching +4. **Auto-process on upload** — no manual script, pipeline runs automatically +5. **Render curated output** — readable, hand-it-to-someone quality + +--- + ## Tech Stack -Uses Lakehouse core — NOT a separate codebase: - **Storage:** Lakehouse storaged (Parquet on object storage) - **Catalog:** Lakehouse catalogd (metadata, PII, lineage) -- **Query:** Lakehouse queryd (DataFusion SQL) +- **Query:** Lakehouse queryd (DataFusion SQL, hot cache, paged results) - **Vectors:** Lakehouse vectord (HNSW search, RAG) -- **AI:** Lakehouse aibridge → Ollama -- **New:** Upload portal (standalone HTML/JS — NOT Dioxus WASM) - -## Why HTML/JS for the Upload Portal - -- No 39MB WASM payload -- Chunked file upload (XMLHttpRequest with progress) -- Works on any browser -- Fast to iterate -- The Dioxus UI can coexist for the SQL/Ask features - ---- - -## Phases - -### Phase 1: Upload Portal -- Chunked file upload (handles 100MB+ PDFs) -- Upload progress bar per file -- Batch upload (drag & drop multiple) -- File list with status (uploading, processing, done, failed) -- Served as static HTML from the gateway - -### Phase 2: Extraction Pipeline -- ChatGPT PDF parser (conversation turns, titles, roles) -- Claude PDF parser (similar structure) -- Generic PDF fallback (page-based) -- Extraction preview: see what was extracted before committing -- Auto-embed extracted turns - -### Phase 3: Knowledge Browser -- Search across all conversations by meaning -- Filter by date, topic, source -- View individual conversations with turns -- Highlight matching content in search results - -### Phase 4: Comparison Tools -- Side-by-side conversation comparison -- Topic tracking across conversations over time -- "Best approach" extraction from successful conversations -- Playbook builder: curate successful patterns - ---- - -## Data Model - -```sql --- Conversations (one per PDF/chat export) -conversations: - conversation_id TEXT PRIMARY KEY - title TEXT - source TEXT -- "chatgpt", "claude", "other" - source_file TEXT - date TEXT - turn_count INTEGER - total_chars INTEGER - topics TEXT -- pipe-separated - summary TEXT -- AI-generated - --- Individual turns -turns: - turn_id TEXT PRIMARY KEY - conversation_id TEXT - turn_number INTEGER - role TEXT -- "user", "assistant" - content TEXT - char_count INTEGER - has_code BOOLEAN - page_number INTEGER - --- Vector embeddings (via vectord) --- Stored in vectors/observer_kb.parquet -``` - ---- - -## Upload Requirements - -- Max file size: 500MB per PDF -- Chunked upload: 5MB chunks with resume on failure -- Batch: up to 50 files at once -- Progress: per-file percentage + overall -- Dedup: skip already-ingested files (content hash) +- **AI Processing:** devop.live LLM Team API (20 modes, multiple models) +- **AI Runtime:** Ollama (local models — qwen2.5, mistral, gemma2, nomic-embed-text) +- **UI:** Pure HTML/JS on :3400 --- ## Non-Goals (for now) -- Live chat ingestion (only PDF exports) -- Editing conversations -- Multi-user access control -- Real-time collaboration +- Real-time conversation ingestion (PDF exports only) +- Multi-user collaboration +- Cloud deployment +- Editing extracted knowledge from UI (read-only for now)