Observer KB: 477 pages from 8 ChatGPT PDFs indexed and searchable

Browse Source

- Fixed Unicode crash in chunker (ligatures like fi causing byte boundary panic)
- Chunker now works with char indices, not byte indices
- 8 PDFs ingested: hybrid_ai_data_lake (133p), spark_vs_flink_iceberg (124p),
  code_sorting (70p), hybrid_indexing (69p), rust_in_tech_stack (49p), etc.
- 477 chunks embedded via supervisor (43/sec)
- HNSW index built in 0.44s
- Search results <200ms, finding correct conversations by meaning
- Tested: "S3 architecture", "CRM WordPress", "Rust vs Python" — all correct

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

...

This commit is contained in:

root

2026-03-27 22:53:27 -05:00

co-authored by Claude Opus 4.6

parent c49b30053b

commit 8ca4ba5717

25 changed files with 657 additions and 24 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

data/datasets/code_sorting_assistance.parquet

BIN

Binary file not shown.

data/datasets/crm_development_options.parquet

BIN

Binary file not shown.

data/datasets/hybrid_ai_data_lake.parquet

BIN

Binary file not shown.

data/datasets/hybrid_indexing_setup.parquet

BIN

Binary file not shown.

data/datasets/legacy_staffing_crm_search.parquet

BIN

Binary file not shown.

data/datasets/llms3_update_prompt.parquet

BIN

Binary file not shown.

data/datasets/rust_in_tech_stack.parquet

BIN

Binary file not shown.

data/datasets/spark_vs_flink_iceberg.parquet

BIN

Binary file not shown.