rootandClaude Opus 4.6
bd68b662b7
Observer KB PRD v2: - Multi-pass pipeline using devop.live's 20 analysis modes - extract → validate → hallucination-check → consensus → organize → render - Q&A chain extraction (questions as anchors) - Curated document output (hand-it-to-someone quality) - Integration architecture with LLM Team API on :5000 Memory updated with: - Two-project structure (Lakehouse core + Observer KB) - What's built vs what's next - Critical: devop.live modes ARE the multi-pass pipeline - Next session: integrate Observer KB with LLM Team modes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
184 lines
6.7 KiB
Markdown
184 lines
6.7 KiB
Markdown
# PRD: Observer Knowledge Base
|
|
|
|
**Status:** Active — extraction pipeline proven, needs multi-pass processing via LLM Team modes
|
|
**Created:** 2026-03-28
|
|
**Owner:** J
|
|
**Branch:** `observer-kb`
|
|
**Parent:** Lakehouse core (crates/)
|
|
|
|
---
|
|
|
|
## Problem
|
|
|
|
AI conversations generate massive amounts of knowledge that's lost to context windows, hallucination, and scope drift. Current approach (single-pass extraction into categories) produces disorganized, hard-to-validate output. Reading raw pages is not useful — the system needs to process, clean, structure, and curate knowledge like an editorial team.
|
|
|
|
## Vision
|
|
|
|
A publishing pipeline that takes raw AI conversation exports and produces well-curated, validated knowledge documents. The output should be something you hand to a team — organized, filtered, and clearly presented. Questions anchor the content, with branching connections showing what was asked, what was answered, and how reasoning was directed.
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
### Multi-Pass Processing via LLM Team Modes
|
|
|
|
The key insight: devop.live already has 20 analysis modes that process content through structured workflows. The Observer KB should route uploaded data through these modes automatically.
|
|
|
|
```
|
|
Raw PDF Upload
|
|
│
|
|
▼ Mode: EXTRACT
|
|
Structured facts, entities, relationships
|
|
│
|
|
▼ Mode: VALIDATOR
|
|
Fact-check claims, score 1-10 confidence
|
|
│
|
|
▼ Mode: HALLUCINATION
|
|
Verify each claim, flag likely hallucinations
|
|
│
|
|
▼ Mode: CONSENSUS
|
|
Multiple passes iterate until extraction converges
|
|
│
|
|
▼ Mode: REDTEAM
|
|
Find flaws in knowledge, patch them
|
|
│
|
|
▼ Mode: PIPELINE (custom)
|
|
Clean → Q&A structure → topic grouping → value ranking
|
|
│
|
|
▼ RENDER
|
|
Well-curated document: questions as anchors, branching connections,
|
|
what was asked → answered → directed
|
|
```
|
|
|
|
### Integration with devop.live (LLM Team)
|
|
|
|
The Observer KB calls the LLM Team API to run modes against extracted content:
|
|
|
|
| Mode | Purpose in KB |
|
|
|---|---|
|
|
| **extract** | Structured facts, entities, relationships from raw text |
|
|
| **validator** | Fact-check extracted claims, confidence scores |
|
|
| **hallucination** | Verify each factual claim independently |
|
|
| **drift** | Process N times, find consistent vs uncertain content |
|
|
| **redteam** | Find flaws in extracted knowledge, improve |
|
|
| **consensus** | Iterate until multiple passes agree on correct extraction |
|
|
| **pipeline** | Chain: clean → extract → validate → organize → render |
|
|
| **research** | Generate follow-up questions, research in parallel |
|
|
| **blindassembly** | Split content into parts, process independently, reassemble |
|
|
|
|
### Data Flow
|
|
|
|
```
|
|
Upload Portal (Observer KB :3400)
|
|
│
|
|
├── PDF → Lakehouse ingestd (text extraction, Parquet storage)
|
|
│
|
|
▼
|
|
Processing Pipeline
|
|
│
|
|
├── Call devop.live API (:5000) with each mode
|
|
├── Each mode processes content through local Ollama models
|
|
├── Results stored back in Lakehouse (Parquet + catalog)
|
|
│
|
|
▼
|
|
Knowledge Store
|
|
│
|
|
├── knowledge_facts: verified claims with confidence scores
|
|
├── knowledge_qa: question → answer pairs with context
|
|
├── knowledge_topics: auto-generated categories
|
|
├── knowledge_connections: how topics relate across conversations
|
|
├── knowledge_contradictions: flagged conflicts between conversations
|
|
├── knowledge_validated: claims that passed hallucination check
|
|
│
|
|
▼
|
|
Rendering
|
|
│
|
|
├── Structured knowledge view (web UI)
|
|
├── Exportable curated document (PDF/markdown)
|
|
└── Question-anchored navigation (Q → A → follow-up → related)
|
|
```
|
|
|
|
---
|
|
|
|
## Processing Pipeline Detail
|
|
|
|
### Pass 1: Clean + Structure
|
|
- Strip PDF artifacts (headers, footers, page numbers)
|
|
- Identify conversation turns (user/AI)
|
|
- Fix encoding issues (ligatures, special chars)
|
|
- Normalize formatting
|
|
|
|
### Pass 2: Q&A Extraction
|
|
- Identify questions asked by user
|
|
- Map each question to its answer
|
|
- Track follow-up questions and how they branch
|
|
- Build question→answer→follow-up chains
|
|
|
|
### Pass 3: Fact Extraction + Validation
|
|
- Run EXTRACT mode: pull structured facts, entities, relationships
|
|
- Run VALIDATOR mode: score each fact 1-10 confidence
|
|
- Run HALLUCINATION mode: verify claims independently
|
|
- Flag contradictions between conversations
|
|
|
|
### Pass 4: Organize + Curate
|
|
- Run CONSENSUS mode: multiple passes converge on correct categorization
|
|
- Auto-generate topic categories from content (not predefined)
|
|
- Rank content by value (decisions > insights > discussion > noise)
|
|
- Connect related content across conversations
|
|
|
|
### Pass 5: Render
|
|
- Question-anchored view: each question is a node, answers branch from it
|
|
- Topic-grouped view: knowledge organized by auto-generated categories
|
|
- Timeline view: how thinking evolved across conversations
|
|
- Export as clean markdown/PDF
|
|
|
|
---
|
|
|
|
## Current State (as of 2026-03-28)
|
|
|
|
### What's Built
|
|
- PDF ingestion (8 PDFs, 477 pages extracted)
|
|
- Single-pass knowledge extraction (496 topics, 300 decisions, 100 insights)
|
|
- Semantic search via HNSW (27ms over 477 chunks)
|
|
- RAG (question → search → LLM answer with sources)
|
|
- 5-pass deep analysis (corpus overview, patterns, evolution, gaps, synthesis)
|
|
- Observer KB web UI (knowledge-first, not page-first)
|
|
- Upload portal with drag & drop
|
|
|
|
### What's Not Built Yet
|
|
- Integration with devop.live LLM Team modes
|
|
- Multi-pass processing pipeline (extract → validate → hallucination check)
|
|
- Q&A chain extraction (question → answer → follow-up mapping)
|
|
- Contradiction detection between conversations
|
|
- Curated document export (PDF/markdown)
|
|
- Question-anchored navigation
|
|
- Auto-processing on upload (currently manual script)
|
|
|
|
### What Needs to Happen Next Session
|
|
1. **Integrate with devop.live API** — call modes programmatically
|
|
2. **Build the multi-pass pipeline** — extract → validate → hallucinate-check → organize
|
|
3. **Q&A chain extraction** — questions as anchors, answers branching
|
|
4. **Auto-process on upload** — no manual script, pipeline runs automatically
|
|
5. **Render curated output** — readable, hand-it-to-someone quality
|
|
|
|
---
|
|
|
|
## Tech Stack
|
|
|
|
- **Storage:** Lakehouse storaged (Parquet on object storage)
|
|
- **Catalog:** Lakehouse catalogd (metadata, PII, lineage)
|
|
- **Query:** Lakehouse queryd (DataFusion SQL, hot cache, paged results)
|
|
- **Vectors:** Lakehouse vectord (HNSW search, RAG)
|
|
- **AI Processing:** devop.live LLM Team API (20 modes, multiple models)
|
|
- **AI Runtime:** Ollama (local models — qwen2.5, mistral, gemma2, nomic-embed-text)
|
|
- **UI:** Pure HTML/JS on :3400
|
|
|
|
---
|
|
|
|
## Non-Goals (for now)
|
|
|
|
- Real-time conversation ingestion (PDF exports only)
|
|
- Multi-user collaboration
|
|
- Cloud deployment
|
|
- Editing extracted knowledge from UI (read-only for now)
|