ADR-004 + internal/pathway: Mem0 versioned trace substrate
Closes Sprint 2 design-bar work (audit reports/scrum/sprint-backlog.md):
S2.1 — ADR-004 documents the pathway-memory data model
S2.2 — pathway port lands with deterministic fixture corpus
and full test coverage on day one
S2.3 — retired traces are excluded from retrieval (test
passes; would fail without the filter)
Mem0-style operations: Add / AddIdempotent / Update / Revise /
Retire / Get / History / Search. Each operation is a method on
Store; persistence is JSONL append-only with corruption recovery
on Replay.
internal/pathway/types.go Trace + event + SearchFilter + sentinel errors
internal/pathway/store.go in-memory state + RWMutex + ops
internal/pathway/persistor.go JSONL append-only log with replay
internal/pathway/store_test.go 20 test funcs covering all 7
Sprint 2 claim rows + concurrency
internal/pathway/persistor_test.go 6 test funcs covering missing-
file, corruption recovery, long-line
handling, parent-dir auto-create,
apply-error skip behavior
Sprint 2 claim coverage row-by-row:
ADD TestAdd_AssignsUIDAndTimestamps + TestAdd_RejectsInvalidJSON
UPDATE TestUpdate_ReplacesContentSameUID + Update_MissingUID_Errors
REVISE TestRevise_LinksToPredecessorViaHistory +
TestRevise_PredecessorMissing_Errors +
TestRevise_ChainOfThree_BackwardWalk
RETIRE TestRetire_ExcludedFromSearch +
TestRetire_StillAccessibleViaGet +
TestRetire_StillAccessibleViaHistory
HISTORY/cycle TestHistory_CycleDetected (injected via internal map),
TestHistory_PredecessorMissing_TruncatesChain,
TestHistory_UnknownUID_ErrorsClean
REPLAY/dup TestAddIdempotent_IncrementsReplayCount (locks the
"replay preserves original content" rule per ADR-004)
CORRUPTION TestPersistor_CorruptedLines_Skipped +
TestPersistor_ApplyError_Skipped
ROUND-TRIP TestPersistor_RoundTrip locks the full Save → fresh
Store → Load → Stats-match contract
Two real bugs caught during testing:
- Add returned the same *Trace stored in the map, so callers
holding a reference saw later mutations. Fixed: clone before
return (matches Get's contract). Same fix in AddIdempotent
+ Revise.
- Test typo: {"v":different} isn't valid JSON; AddIdempotent's
json.Valid rejected it as ErrInvalidContent. Test fixed to
use {"v":"different"}; the validation behavior is correct.
Skipped this commit (next):
- cmd/pathwayd HTTP binary
- gateway routing for /v1/pathway/*
- end-to-end smoke
These add the wire surface; the substrate ships first so the
wire layer can be a pure proxy in the next commit.
Verified:
go test -count=1 ./internal/pathway/ — 26 tests green
just verify — vet + test + 9 smokes 34s
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
+120
-3
@@ -242,6 +242,123 @@ need rotate-without-restart.
|
||||
|
||||
---
|
||||
|
||||
(Future ADRs from ADR-004 onward will be added as the Go
|
||||
implementation accrues design decisions — e.g. HNSW parameter
|
||||
choices, pathway-memory hash function, auditor model rotation, etc.)
|
||||
## ADR-004: Pathway memory data model — Mem0-style versioned traces
|
||||
**Date:** 2026-04-29
|
||||
**Decided by:** J + Claude
|
||||
**Status:** Decided — substrate landing in `internal/pathway/`
|
||||
|
||||
**Decision:** Pathway memory is an append-only event log of opaque
|
||||
traces with Mem0-style semantics: Add / Update / Revise / Retire /
|
||||
History / Search. Each trace has a UID; revisions chain backward
|
||||
via `predecessor_uid` so the full history is reconstructible.
|
||||
Persistence is JSONL append-only with full-replay on load;
|
||||
corruption recovery skips bad lines without halting startup.
|
||||
|
||||
### Operations
|
||||
|
||||
| Op | Effect |
|
||||
|---|---|
|
||||
| `Add(content, tags...)` | New UID, stored fresh, replay_count=1. |
|
||||
| `AddIdempotent(uid, content, tags...)` | If UID exists → replay_count++. Else → Add with that UID. |
|
||||
| `Update(uid, content)` | In-place content replacement (same UID). Bumps `updated_at_ns`. NOT a revision — same trace, new content. |
|
||||
| `Revise(predecessorUID, content, tags...)` | New UID with `predecessor_uid` set. Old trace stays accessible via History. Failure modes: predecessor missing → error; predecessor retired → still allowed (revisions of retired traces are valid). |
|
||||
| `Retire(uid)` | Sets `retired=true`. Excluded from `Search` by default; still accessible via `Get` and `History`. |
|
||||
| `Get(uid)` | Returns the trace (including if retired); error on missing. |
|
||||
| `History(uid)` | Walks `predecessor_uid` chain backward, returns slice [self, parent, grandparent, ...]. Cycle-detected via visited-set; returns error on cycle (which only happens if persistence file was hand-edited). |
|
||||
| `Search(filter)` | Returns matching traces. Default excludes retired; opt in via `IncludeRetired: true`. Filters: tag-match, content-substring, time range. |
|
||||
|
||||
### Why Mem0-style + Why these specific ops
|
||||
|
||||
- **Mem0** (memory pattern from the OpenAI Memories paper / Mem0 lib)
|
||||
is the canonical "agent memory" interface for the same reason
|
||||
Markdown is the canonical text format: it's the lowest-common-
|
||||
denominator that the entire ecosystem assumes. Adopting it lets
|
||||
agent loops written against any Mem0-aware substrate work here.
|
||||
- Update vs Revise are deliberately separate. Update is "I noticed
|
||||
a typo in my note." Revise is "I now believe something different
|
||||
than I did when I wrote this; preserve the old belief for audit."
|
||||
Conflating them loses the audit trail.
|
||||
- Retire vs Delete is deliberate. Retire stops a trace from
|
||||
surfacing in search but preserves it for history reconstruction.
|
||||
Delete (which we don't expose) would break references.
|
||||
|
||||
### Trace data shape
|
||||
|
||||
```go
|
||||
type Trace struct {
|
||||
UID string // UUID v4 unless caller provides one
|
||||
Content json.RawMessage // opaque, schema is caller's contract
|
||||
PredecessorUID string // empty if root revision
|
||||
CreatedAtNs int64
|
||||
UpdatedAtNs int64
|
||||
Retired bool
|
||||
ReplayCount int // ≥1 for any stored trace
|
||||
Tags []string // for Search
|
||||
}
|
||||
```
|
||||
|
||||
`Content` is opaque JSON (not a struct) so callers can store any
|
||||
shape — the data model doesn't constrain semantics. Callers add
|
||||
their own validators on top.
|
||||
|
||||
### Persistence
|
||||
|
||||
JSONL append-only log under `_pathway/<store_name>.jsonl`. Each
|
||||
mutation appends one JSON line:
|
||||
|
||||
```
|
||||
{"op":"add", "trace":{...}}
|
||||
{"op":"update", "uid":"…", "content":"…"}
|
||||
{"op":"revise", "trace":{…}} # trace.PredecessorUID is set
|
||||
{"op":"retire", "uid":"…"}
|
||||
{"op":"replay", "uid":"…"} # idempotent re-add hit
|
||||
```
|
||||
|
||||
On startup, replay every line in order, building in-memory state.
|
||||
A malformed line logs a warn and is skipped; load continues.
|
||||
Corruption tolerance is non-optional — partial state is better
|
||||
than no state for an agent substrate.
|
||||
|
||||
Compaction is a future concern. A 100K-trace log replays in
|
||||
seconds; below that scale, JSONL append is the simplest correct
|
||||
choice. When compaction lands, the format will be: snapshot file
|
||||
(full state JSON) + tail JSONL since snapshot. Detect snapshot,
|
||||
load it, then replay tail.
|
||||
|
||||
### Cycle safety
|
||||
|
||||
UIDs are generated server-side via `uuid.New()` (existing dep —
|
||||
catalogd uses it). New UID for every Add and Revise. The data
|
||||
model itself can't form cycles — every Revise points at an
|
||||
EXISTING uid, and the new uid didn't exist a moment ago.
|
||||
|
||||
History walks defensively anyway: visited-set tracks UIDs seen
|
||||
this walk; if we encounter a duplicate, return error. Protects
|
||||
against corruption (manual edit, bug in a future op) without
|
||||
constraining the happy path.
|
||||
|
||||
### Storage location
|
||||
|
||||
JSONL file path is configurable per store. Default:
|
||||
`/var/lib/lakehouse/pathway/<name>.jsonl` for prod; tests use
|
||||
`t.TempDir()`. Persistence is OPTIONAL — empty path means
|
||||
in-memory only (matches vectord G1's pattern).
|
||||
|
||||
### What this ADR does NOT do
|
||||
|
||||
- **No HTTP surface decision.** Whether `cmd/pathwayd` is its own
|
||||
binary or routes get added to `cmd/vectord` is the next ADR's
|
||||
concern. The substrate is a pure library either way.
|
||||
- **No vector index integration.** Pathway traces can carry a
|
||||
vector embedding in `Content` (caller decides), but this ADR
|
||||
doesn't define how the substrate integrates with `vectord`'s
|
||||
HNSW indexes. That's the staffing co-pilot's design problem
|
||||
when those layers compose.
|
||||
- **No agent-loop semantics.** "When does an agent ADD vs
|
||||
REVISE?" is a workflow decision, not a substrate decision.
|
||||
|
||||
---
|
||||
|
||||
(Future ADRs from ADR-005 onward will be added as the Go
|
||||
implementation accrues design decisions — e.g. observer fail-safe
|
||||
semantics, distillation rebuild, gRPC adapter wire format, etc.)
|
||||
|
||||
Reference in New Issue
Block a user