lakehouse/apps/observer-kb/docs/PRD.md

PRD: Observer Knowledge Base

Status: Active — extraction pipeline proven, needs multi-pass processing via LLM Team modes Created: 2026-03-28 Owner: J Branch: observer-kb Parent: Lakehouse core (crates/)

---

Problem

AI conversations generate massive amounts of knowledge that's lost to context windows, hallucination, and scope drift. Current approach (single-pass extraction into categories) produces disorganized, hard-to-validate output. Reading raw pages is not useful — the system needs to process, clean, structure, and curate knowledge like an editorial team.

Vision

A publishing pipeline that takes raw AI conversation exports and produces well-curated, validated knowledge documents. The output should be something you hand to a team — organized, filtered, and clearly presented. Questions anchor the content, with branching connections showing what was asked, what was answered, and how reasoning was directed.

---

Architecture

Multi-Pass Processing via LLM Team Modes

The key insight: devop.live already has 20 analysis modes that process content through structured workflows. The Observer KB should route uploaded data through these modes automatically.

Raw PDF Upload
    │
    ▼ Mode: EXTRACT
    Structured facts, entities, relationships
    │
    ▼ Mode: VALIDATOR
    Fact-check claims, score 1-10 confidence
    │
    ▼ Mode: HALLUCINATION
    Verify each claim, flag likely hallucinations
    │
    ▼ Mode: CONSENSUS
    Multiple passes iterate until extraction converges
    │
    ▼ Mode: REDTEAM
    Find flaws in knowledge, patch them
    │
    ▼ Mode: PIPELINE (custom)
    Clean → Q&A structure → topic grouping → value ranking
    │
    ▼ RENDER
    Well-curated document: questions as anchors, branching connections,
    what was asked → answered → directed

Integration with devop.live (LLM Team)

The Observer KB calls the LLM Team API to run modes against extracted content:

Mode	Purpose in KB
extract	Structured facts, entities, relationships from raw text
validator	Fact-check extracted claims, confidence scores
hallucination	Verify each factual claim independently
drift	Process N times, find consistent vs uncertain content
redteam	Find flaws in extracted knowledge, improve
consensus	Iterate until multiple passes agree on correct extraction
pipeline	Chain: clean → extract → validate → organize → render
research	Generate follow-up questions, research in parallel
blindassembly	Split content into parts, process independently, reassemble

Data Flow

Upload Portal (Observer KB :3400)
    │
    ├── PDF → Lakehouse ingestd (text extraction, Parquet storage)
    │
    ▼
Processing Pipeline
    │
    ├── Call devop.live API (:5000) with each mode
    ├── Each mode processes content through local Ollama models
    ├── Results stored back in Lakehouse (Parquet + catalog)
    │
    ▼
Knowledge Store
    │
    ├── knowledge_facts: verified claims with confidence scores
    ├── knowledge_qa: question → answer pairs with context
    ├── knowledge_topics: auto-generated categories
    ├── knowledge_connections: how topics relate across conversations
    ├── knowledge_contradictions: flagged conflicts between conversations
    ├── knowledge_validated: claims that passed hallucination check
    │
    ▼
Rendering
    │
    ├── Structured knowledge view (web UI)
    ├── Exportable curated document (PDF/markdown)
    └── Question-anchored navigation (Q → A → follow-up → related)

---

Processing Pipeline Detail

Pass 1: Clean + Structure

• Strip PDF artifacts (headers, footers, page numbers)
• Identify conversation turns (user/AI)
• Fix encoding issues (ligatures, special chars)
• Normalize formatting

Pass 2: Q&A Extraction

• Identify questions asked by user
• Map each question to its answer
• Track follow-up questions and how they branch
• Build question→answer→follow-up chains

Pass 3: Fact Extraction + Validation

• Run EXTRACT mode: pull structured facts, entities, relationships
• Run VALIDATOR mode: score each fact 1-10 confidence
• Run HALLUCINATION mode: verify claims independently
• Flag contradictions between conversations

Pass 4: Organize + Curate

• Run CONSENSUS mode: multiple passes converge on correct categorization
• Auto-generate topic categories from content (not predefined)
• Rank content by value (decisions > insights > discussion > noise)
• Connect related content across conversations

Pass 5: Render

• Question-anchored view: each question is a node, answers branch from it
• Topic-grouped view: knowledge organized by auto-generated categories
• Timeline view: how thinking evolved across conversations
• Export as clean markdown/PDF

---

Current State (as of 2026-03-28)

What's Built

• PDF ingestion (8 PDFs, 477 pages extracted)
• Single-pass knowledge extraction (496 topics, 300 decisions, 100 insights)
• Semantic search via HNSW (27ms over 477 chunks)
• RAG (question → search → LLM answer with sources)
• 5-pass deep analysis (corpus overview, patterns, evolution, gaps, synthesis)
• Observer KB web UI (knowledge-first, not page-first)
• Upload portal with drag & drop

What's Not Built Yet

• Integration with devop.live LLM Team modes
• Multi-pass processing pipeline (extract → validate → hallucination check)
• Q&A chain extraction (question → answer → follow-up mapping)
• Contradiction detection between conversations
• Curated document export (PDF/markdown)
• Question-anchored navigation
• Auto-processing on upload (currently manual script)

What Needs to Happen Next Session

1. Integrate with devop.live API — call modes programmatically
2. Build the multi-pass pipeline — extract → validate → hallucinate-check → organize
3. Q&A chain extraction — questions as anchors, answers branching
4. Auto-process on upload — no manual script, pipeline runs automatically
5. Render curated output — readable, hand-it-to-someone quality

---

Tech Stack

• Storage: Lakehouse storaged (Parquet on object storage)
• Catalog: Lakehouse catalogd (metadata, PII, lineage)
• Query: Lakehouse queryd (DataFusion SQL, hot cache, paged results)
• Vectors: Lakehouse vectord (HNSW search, RAG)
• AI Processing: devop.live LLM Team API (20 modes, multiple models)
• AI Runtime: Ollama (local models — qwen2.5, mistral, gemma2, nomic-embed-text)
• UI: Pure HTML/JS on :3400

---

Non-Goals (for now)

• Real-time conversation ingestion (PDF exports only)
• Multi-user collaboration
• Cloud deployment
• Editing extracted knowledge from UI (read-only for now)