Metacognitive knowledge extraction: 496 topics, 300 decisions, 100 insights
- extract_knowledge.py: LLM processes sampled pages from all conversations - Extracts structured JSON: topics, decisions, insights, open questions - Stores as queryable Parquet tables (knowledge_topics, knowledge_decisions, knowledge_insights, knowledge_questions) - Each finding traces to source conversation + page number - Fixed Unicode chunker crash (byte boundary on multi-byte chars) - Tested on 8 real ChatGPT PDFs, 477 pages: 496 topics, 300 decisions, 100 insights, 200 open questions extracted Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"id": "fe6b6872-03f8-491a-b2c2-7d25b19cd9bb",
|
||||
"name": "knowledge_decisions",
|
||||
"schema_fingerprint": "be65fb1ba2121bcf43c195c06dbdc0ae892eb835f4a909e9c1c0438ac81a1d0d",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"key": "datasets/knowledge_decisions.parquet",
|
||||
"size_bytes": 21274,
|
||||
"created_at": "2026-03-28T04:06:05.360578855Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-28T04:06:05.360579258Z",
|
||||
"updated_at": "2026-03-28T04:06:05.360801969Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "conversation",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "page",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "decision",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "csv",
|
||||
"source_file": "knowledge_decisions.csv",
|
||||
"ingest_job": "ingest-1774670765360",
|
||||
"ingest_timestamp": "2026-03-28T04:06:05.360578855Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 300
|
||||
}
|
||||
Reference in New Issue
Block a user