# PRD: Observer Knowledge Base

**Status:** Active — extraction pipeline proven, needs multi-pass processing via LLM Team modes
**Created:** 2026-03-28
**Owner:** J
**Branch:** `observer-kb`
**Parent:** Lakehouse core (crates/)

---

## Problem

AI conversations generate massive amounts of knowledge that's lost to context windows, hallucination, and scope drift. Current approach (single-pass extraction into categories) produces disorganized, hard-to-validate output. Reading raw pages is not useful — the system needs to process, clean, structure, and curate knowledge like an editorial team.

## Vision

A publishing pipeline that takes raw AI conversation exports and produces well-curated, validated knowledge documents. The output should be something you hand to a team — organized, filtered, and clearly presented. Questions anchor the content, with branching connections showing what was asked, what was answered, and how reasoning was directed.

---

## Architecture

### Multi-Pass Processing via LLM Team Modes

The key insight: devop.live already has 20 analysis modes that process content through structured workflows. The Observer KB should route uploaded data through these modes automatically.

```
Raw PDF Upload
    │
    ▼ Mode: EXTRACT
    Structured facts, entities, relationships
    │
    ▼ Mode: VALIDATOR
    Fact-check claims, score 1-10 confidence
    │
    ▼ Mode: HALLUCINATION
    Verify each claim, flag likely hallucinations
    │
    ▼ Mode: CONSENSUS
    Multiple passes iterate until extraction converges
    │
    ▼ Mode: REDTEAM
    Find flaws in knowledge, patch them
    │
    ▼ Mode: PIPELINE (custom)
    Clean → Q&A structure → topic grouping → value ranking
    │
    ▼ RENDER
    Well-curated document: questions as anchors, branching connections,
    what was asked → answered → directed
```

### Integration with devop.live (LLM Team)

The Observer KB calls the LLM Team API to run modes against extracted content:

| Mode | Purpose in KB |
|---|---|
| **extract** | Structured facts, entities, relationships from raw text |
| **validator** | Fact-check extracted claims, confidence scores |
| **hallucination** | Verify each factual claim independently |
| **drift** | Process N times, find consistent vs uncertain content |
| **redteam** | Find flaws in extracted knowledge, improve |
| **consensus** | Iterate until multiple passes agree on correct extraction |
| **pipeline** | Chain: clean → extract → validate → organize → render |
| **research** | Generate follow-up questions, research in parallel |
| **blindassembly** | Split content into parts, process independently, reassemble |

### Data Flow

```
Upload Portal (Observer KB :3400)
    │
    ├── PDF → Lakehouse ingestd (text extraction, Parquet storage)
    │
    ▼
Processing Pipeline
    │
    ├── Call devop.live API (:5000) with each mode
    ├── Each mode processes content through local Ollama models
    ├── Results stored back in Lakehouse (Parquet + catalog)
    │
    ▼
Knowledge Store
    │
    ├── knowledge_facts: verified claims with confidence scores
    ├── knowledge_qa: question → answer pairs with context
    ├── knowledge_topics: auto-generated categories
    ├── knowledge_connections: how topics relate across conversations
    ├── knowledge_contradictions: flagged conflicts between conversations
    ├── knowledge_validated: claims that passed hallucination check
    │
    ▼
Rendering
    │
    ├── Structured knowledge view (web UI)
    ├── Exportable curated document (PDF/markdown)
    └── Question-anchored navigation (Q → A → follow-up → related)
```

---

## Processing Pipeline Detail

### Pass 1: Clean + Structure
- Strip PDF artifacts (headers, footers, page numbers)
- Identify conversation turns (user/AI)
- Fix encoding issues (ligatures, special chars)
- Normalize formatting

### Pass 2: Q&A Extraction
- Identify questions asked by user
- Map each question to its answer
- Track follow-up questions and how they branch
- Build question→answer→follow-up chains

### Pass 3: Fact Extraction + Validation
- Run EXTRACT mode: pull structured facts, entities, relationships
- Run VALIDATOR mode: score each fact 1-10 confidence
- Run HALLUCINATION mode: verify claims independently
- Flag contradictions between conversations

### Pass 4: Organize + Curate
- Run CONSENSUS mode: multiple passes converge on correct categorization
- Auto-generate topic categories from content (not predefined)
- Rank content by value (decisions > insights > discussion > noise)
- Connect related content across conversations

### Pass 5: Render
- Question-anchored view: each question is a node, answers branch from it
- Topic-grouped view: knowledge organized by auto-generated categories
- Timeline view: how thinking evolved across conversations
- Export as clean markdown/PDF

---

## Current State (as of 2026-03-28)

### What's Built
- PDF ingestion (8 PDFs, 477 pages extracted)
- Single-pass knowledge extraction (496 topics, 300 decisions, 100 insights)
- Semantic search via HNSW (27ms over 477 chunks)
- RAG (question → search → LLM answer with sources)
- 5-pass deep analysis (corpus overview, patterns, evolution, gaps, synthesis)
- Observer KB web UI (knowledge-first, not page-first)
- Upload portal with drag & drop

### What's Not Built Yet
- Integration with devop.live LLM Team modes
- Multi-pass processing pipeline (extract → validate → hallucination check)
- Q&A chain extraction (question → answer → follow-up mapping)
- Contradiction detection between conversations
- Curated document export (PDF/markdown)
- Question-anchored navigation
- Auto-processing on upload (currently manual script)

### What Needs to Happen Next Session
1. **Integrate with devop.live API** — call modes programmatically
2. **Build the multi-pass pipeline** — extract → validate → hallucinate-check → organize
3. **Q&A chain extraction** — questions as anchors, answers branching
4. **Auto-process on upload** — no manual script, pipeline runs automatically
5. **Render curated output** — readable, hand-it-to-someone quality

---

## Tech Stack

- **Storage:** Lakehouse storaged (Parquet on object storage)
- **Catalog:** Lakehouse catalogd (metadata, PII, lineage)
- **Query:** Lakehouse queryd (DataFusion SQL, hot cache, paged results)
- **Vectors:** Lakehouse vectord (HNSW search, RAG)
- **AI Processing:** devop.live LLM Team API (20 modes, multiple models)
- **AI Runtime:** Ollama (local models — qwen2.5, mistral, gemma2, nomic-embed-text)
- **UI:** Pure HTML/JS on :3400

---

## Non-Goals (for now)

- Real-time conversation ingestion (PDF exports only)
- Multi-user collaboration
- Cloud deployment
- Editing extracted knowledge from UI (read-only for now)