Update PRD + memory: Observer KB needs devop.live LLM Team mode integration

Observer KB PRD v2: - Multi-pass pipeline using devop.live's 20 analysis modes - extract → validate → hallucination-check → consensus → organize → render - Q&A chain extraction (questions as anchors) - Curated document output (hand-it-to-someone quality) - Integration architecture with LLM Team API on :5000 Memory updated with: - Two-project structure (Lakehouse core + Observer KB) - What's built vs what's next - Critical: devop.live modes ARE the multi-pass pipeline - Next session: integrate Observer KB with LLM Team modes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 23:35:15 -05:00
parent 3cf000ccf3
commit bd68b662b7
1 changed files with 145 additions and 121 deletions
@@ -1,6 +1,6 @@
 # PRD: Observer Knowledge Base

-**Status:** Active
+**Status:** Active — extraction pipeline proven, needs multi-pass processing via LLM Team modes
 **Created:** 2026-03-28
 **Owner:** J
 **Branch:** `observer-kb`
@@ -10,150 +10,174 @@

 ## Problem

-AI conversations (ChatGPT, Claude, etc.) generate massive amounts of knowledge — decisions, research, code, strategies, troubleshooting — that disappear into PDF exports and chat logs. There's no way to:
- Search across hundreds of conversations by meaning
- Find "what did I discuss about X six months ago?"
- Compare approaches across different conversations
- Build a reusable knowledge base from past AI interactions
- Track which ideas worked and which didn't
+AI conversations generate massive amounts of knowledge that's lost to context windows, hallucination, and scope drift. Current approach (single-pass extraction into categories) produces disorganized, hard-to-validate output. Reading raw pages is not useful — the system needs to process, clean, structure, and curate knowledge like an editorial team.

-## Solution
+## Vision

-An upload portal + knowledge extraction pipeline built on the Lakehouse infrastructure. Upload ChatGPT/Claude PDFs → extract conversations → chunk and embed → searchable knowledge base with comparison tools.
+A publishing pipeline that takes raw AI conversation exports and produces well-curated, validated knowledge documents. The output should be something you hand to a team — organized, filtered, and clearly presented. Questions anchor the content, with branching connections showing what was asked, what was answered, and how reasoning was directed.

 ---

 ## Architecture

+### Multi-Pass Processing via LLM Team Modes
+
+The key insight: devop.live already has 20 analysis modes that process content through structured workflows. The Observer KB should route uploaded data through these modes automatically.
+
 ```
-Upload Portal (web UI)
+Raw PDF Upload
    │
-    ├── Upload large PDFs (chunked upload for big files)
-    ├── Progress tracking per file
-    └── Visual extraction preview
+    ▼ Mode: EXTRACT
+    Structured facts, entities, relationships
    │
-    ▼
-Extraction Pipeline (uses Lakehouse ingestd)
+    ▼ Mode: VALIDATOR
+    Fact-check claims, score 1-10 confidence
    │
-    ├── PDF text extraction (lopdf)
-    ├── Conversation turn parsing (user/assistant)
-    ├── Topic detection
-    ├── Metadata extraction (dates, titles, participants)
-    └── Quality scoring (how useful was this conversation?)
+    ▼ Mode: HALLUCINATION
+    Verify each claim, flag likely hallucinations
    │
-    ▼
-Lakehouse Storage
+    ▼ Mode: CONSENSUS
+    Multiple passes iterate until extraction converges
    │
-    ├── conversations table (id, title, source, date, turns, topic)
-    ├── turns table (conversation_id, turn_num, role, content)
-    ├── topics table (topic, conversation_ids, summary)
-    └── vectors/ (embedded turns for semantic search)
+    ▼ Mode: REDTEAM
+    Find flaws in knowledge, patch them
    │
-    ▼
-Knowledge Interface
+    ▼ Mode: PIPELINE (custom)
+    Clean → Q&A structure → topic grouping → value ranking
    │
-    ├── Search: "what did I discuss about Kubernetes?"
-    ├── Compare: conversation A vs B on same topic
-    ├── Timeline: when did I first explore this idea?
-    ├── Extract: pull key decisions/code/strategies
-    └── Playbook: which approaches worked?
+    ▼ RENDER
+    Well-curated document: questions as anchors, branching connections,
+    what was asked → answered → directed
 ```

+### Integration with devop.live (LLM Team)
+
+The Observer KB calls the LLM Team API to run modes against extracted content:
+
+| Mode | Purpose in KB |
+|---|---|
+| **extract** | Structured facts, entities, relationships from raw text |
+| **validator** | Fact-check extracted claims, confidence scores |
+| **hallucination** | Verify each factual claim independently |
+| **drift** | Process N times, find consistent vs uncertain content |
+| **redteam** | Find flaws in extracted knowledge, improve |
+| **consensus** | Iterate until multiple passes agree on correct extraction |
+| **pipeline** | Chain: clean → extract → validate → organize → render |
+| **research** | Generate follow-up questions, research in parallel |
+| **blindassembly** | Split content into parts, process independently, reassemble |
+
+### Data Flow
+
+```
+Upload Portal (Observer KB :3400)
+    │
+    ├── PDF → Lakehouse ingestd (text extraction, Parquet storage)
+    │
+    ▼
+Processing Pipeline
+    │
+    ├── Call devop.live API (:5000) with each mode
+    ├── Each mode processes content through local Ollama models
+    ├── Results stored back in Lakehouse (Parquet + catalog)
+    │
+    ▼
+Knowledge Store
+    │
+    ├── knowledge_facts: verified claims with confidence scores
+    ├── knowledge_qa: question → answer pairs with context
+    ├── knowledge_topics: auto-generated categories
+    ├── knowledge_connections: how topics relate across conversations
+    ├── knowledge_contradictions: flagged conflicts between conversations
+    ├── knowledge_validated: claims that passed hallucination check
+    │
+    ▼
+Rendering
+    │
+    ├── Structured knowledge view (web UI)
+    ├── Exportable curated document (PDF/markdown)
+    └── Question-anchored navigation (Q → A → follow-up → related)
+```
+
+---
+
+## Processing Pipeline Detail
+
+### Pass 1: Clean + Structure
+- Strip PDF artifacts (headers, footers, page numbers)
+- Identify conversation turns (user/AI)
+- Fix encoding issues (ligatures, special chars)
+- Normalize formatting
+
+### Pass 2: Q&A Extraction
+- Identify questions asked by user
+- Map each question to its answer
+- Track follow-up questions and how they branch
+- Build question→answer→follow-up chains
+
+### Pass 3: Fact Extraction + Validation
+- Run EXTRACT mode: pull structured facts, entities, relationships
+- Run VALIDATOR mode: score each fact 1-10 confidence
+- Run HALLUCINATION mode: verify claims independently
+- Flag contradictions between conversations
+
+### Pass 4: Organize + Curate
+- Run CONSENSUS mode: multiple passes converge on correct categorization
+- Auto-generate topic categories from content (not predefined)
+- Rank content by value (decisions > insights > discussion > noise)
+- Connect related content across conversations
+
+### Pass 5: Render
+- Question-anchored view: each question is a node, answers branch from it
+- Topic-grouped view: knowledge organized by auto-generated categories
+- Timeline view: how thinking evolved across conversations
+- Export as clean markdown/PDF
+
+---
+
+## Current State (as of 2026-03-28)
+
+### What's Built
+- PDF ingestion (8 PDFs, 477 pages extracted)
+- Single-pass knowledge extraction (496 topics, 300 decisions, 100 insights)
+- Semantic search via HNSW (27ms over 477 chunks)
+- RAG (question → search → LLM answer with sources)
+- 5-pass deep analysis (corpus overview, patterns, evolution, gaps, synthesis)
+- Observer KB web UI (knowledge-first, not page-first)
+- Upload portal with drag & drop
+
+### What's Not Built Yet
+- Integration with devop.live LLM Team modes
+- Multi-pass processing pipeline (extract → validate → hallucination check)
+- Q&A chain extraction (question → answer → follow-up mapping)
+- Contradiction detection between conversations
+- Curated document export (PDF/markdown)
+- Question-anchored navigation
+- Auto-processing on upload (currently manual script)
+
+### What Needs to Happen Next Session
+1. **Integrate with devop.live API** — call modes programmatically
+2. **Build the multi-pass pipeline** — extract → validate → hallucinate-check → organize
+3. **Q&A chain extraction** — questions as anchors, answers branching
+4. **Auto-process on upload** — no manual script, pipeline runs automatically
+5. **Render curated output** — readable, hand-it-to-someone quality
+
+---
+
 ## Tech Stack

-Uses Lakehouse core — NOT a separate codebase:
 - **Storage:** Lakehouse storaged (Parquet on object storage)
 - **Catalog:** Lakehouse catalogd (metadata, PII, lineage)
- **Query:** Lakehouse queryd (DataFusion SQL)
+- **Query:** Lakehouse queryd (DataFusion SQL, hot cache, paged results)
 - **Vectors:** Lakehouse vectord (HNSW search, RAG)
- **AI:** Lakehouse aibridge → Ollama
- **New:** Upload portal (standalone HTML/JS — NOT Dioxus WASM)
-
-## Why HTML/JS for the Upload Portal
-
- No 39MB WASM payload
- Chunked file upload (XMLHttpRequest with progress)
- Works on any browser
- Fast to iterate
- The Dioxus UI can coexist for the SQL/Ask features
-
---
-
-## Phases
-
-### Phase 1: Upload Portal
- Chunked file upload (handles 100MB+ PDFs)
- Upload progress bar per file
- Batch upload (drag & drop multiple)
- File list with status (uploading, processing, done, failed)
- Served as static HTML from the gateway
-
-### Phase 2: Extraction Pipeline
- ChatGPT PDF parser (conversation turns, titles, roles)
- Claude PDF parser (similar structure)
- Generic PDF fallback (page-based)
- Extraction preview: see what was extracted before committing
- Auto-embed extracted turns
-
-### Phase 3: Knowledge Browser
- Search across all conversations by meaning
- Filter by date, topic, source
- View individual conversations with turns
- Highlight matching content in search results
-
-### Phase 4: Comparison Tools
- Side-by-side conversation comparison
- Topic tracking across conversations over time
- "Best approach" extraction from successful conversations
- Playbook builder: curate successful patterns
-
---
-
-## Data Model
-
-```sql
-- Conversations (one per PDF/chat export)
-conversations:
-  conversation_id TEXT PRIMARY KEY
-  title TEXT
-  source TEXT              -- "chatgpt", "claude", "other"
-  source_file TEXT
-  date TEXT
-  turn_count INTEGER
-  total_chars INTEGER
-  topics TEXT              -- pipe-separated
-  summary TEXT             -- AI-generated
-
-- Individual turns
-turns:
-  turn_id TEXT PRIMARY KEY
-  conversation_id TEXT
-  turn_number INTEGER
-  role TEXT                -- "user", "assistant"
-  content TEXT
-  char_count INTEGER
-  has_code BOOLEAN
-  page_number INTEGER
-
-- Vector embeddings (via vectord)
-- Stored in vectors/observer_kb.parquet
-```
-
---
-
-## Upload Requirements
-
- Max file size: 500MB per PDF
- Chunked upload: 5MB chunks with resume on failure
- Batch: up to 50 files at once
- Progress: per-file percentage + overall
- Dedup: skip already-ingested files (content hash)
+- **AI Processing:** devop.live LLM Team API (20 modes, multiple models)
+- **AI Runtime:** Ollama (local models — qwen2.5, mistral, gemma2, nomic-embed-text)
+- **UI:** Pure HTML/JS on :3400

 ---

 ## Non-Goals (for now)

- Live chat ingestion (only PDF exports)
- Editing conversations
- Multi-user access control
- Real-time collaboration
+- Real-time conversation ingestion (PDF exports only)
+- Multi-user collaboration
+- Cloud deployment
+- Editing extracted knowledge from UI (read-only for now)