Phase 21 Rust port + Phase 27 playbook versioning + doc-sync

Phase 21 — Rust port of scratchpad + tree-split primitives (companion to
the 2026-04-21 TS shipment). New crates/aibridge modules:

  context.rs       — estimate_tokens (chars/4 ceil), context_window_for,
                     assert_context_budget returning a BudgetCheck with
                     numeric diagnostics on both success and overflow.
                     Windows table mirrors config/models.json.
  continuation.rs  — generate_continuable<G: TextGenerator>. Handles the
                     two failure modes: empty-response from thinking
                     models (geometric 2x budget backoff up to budget_cap)
                     and truncated-non-empty (continuation with partial
                     as scratchpad). is_structurally_complete balances
                     braces then JSON.parse-checks. Guards the degen case
                     "all retries empty, don't loop on empty partial".
  tree_split.rs    — generate_tree_split map->reduce with running
                     scratchpad. Per-shard + reduce-prompt go through
                     assert_context_budget first; loud-fails rather than
                     silently truncating. Oldest-digest-first scratchpad
                     truncation at scratchpad_budget (default 6000 t).

TextGenerator trait (native async-fn-in-trait, edition 2024). AiClient
implements it; ScriptedGenerator test double lets tests inject canned
sequences without a live Ollama.

GenerateRequest gained think: Option<bool> — forwards to sidecar for
per-call hidden-reasoning opt-out on hot-path JSON emitters. Three
existing callsites updated (rag.rs x2, service.rs hybrid answer).

Phase 27 — Playbook versioning. PlaybookEntry gained four optional
fields (all #[serde(default)] so pre-Phase-27 state loads as roots):

  version           u32, default 1
  parent_id         Option<String>, previous version's playbook_id
  superseded_at     Option<String>, set when newer version replaces
  superseded_by     Option<String>, the playbook_id that replaced

New methods:

  revise_entry(parent_id, new_entry) — appends new version, stamps
    superseded_at+superseded_by on parent, inherits parent_id and sets
    version = parent + 1 on the new entry. Rejects revising a retired
    or already-superseded parent (tip-of-chain is the only valid
    revise target).
  history(playbook_id) — returns full chain root->tip from any node.
    Walks parent_id back to root, then superseded_by forward to tip.
    Cycle-safe.

Superseded entries excluded from boost (same rule as retired): filter
in compute_boost_for_filtered_with_role (both active-entries prefilter
and geo-filtered path), rebuild_geo_index, and upsert_entry's existing-
idx search. status_counts returns (total, retired, superseded, failures);
/status JSON reports active = total - retired - superseded.

Endpoints:
  POST /vectors/playbook_memory/revise
  GET  /vectors/playbook_memory/history/{id}

Doc-sync — PHASES.md + PRD.md drifted from git after Phases 24-26
shipped. Fixes applied:

  - Phase 24 marked shipped (commit b95dd86) with detail of observer
    HTTP ingest + scenario outcome streaming. PRD "NOT YET WIRED"
    rewritten to reflect shipped state.
  - Phase 25 (validity windows, commit e0a843d) added to PHASES +
    PRD.
  - Phase 26 (Mem0 upsert + Letta hot cache, commit 640db8c) added.
  - Phase 27 entry added to both docs.
  - Phase 19.6 time decay corrected: was documented as "deferred",
    actually wired via BOOST_HALF_LIFE_DAYS = 30.0 in playbook_memory.rs.
  - Phase E/Phase 8 tombstone-at-compaction limit note updated —
    Phase E.2 closed it.

Tests: 8 new version_tests in vectord (chain-metadata stamping,
retired/superseded parent rejection, boost exclusion, history from
root/tip/middle, legacy default round-trip, status counts). 25 new
aibridge tests (context/continuation/tree_split). Workspace total
145 green (was 120).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
profit
2026-04-21 17:40:49 -05:00
co-authored by Claude Opus 4.7
parent 640db8c63c
commit a6f12e2609
10 changed files with 1506 additions and 29 deletions
+347 -8
View File
@@ -113,8 +113,31 @@ pub struct PlaybookEntry {
/// "manual: operator requested via POST /retire"
#[serde(default)]
pub retirement_reason: Option<String>,
/// Phase 27 — monotonic version counter within a playbook chain.
/// First version is 1; `revise_entry` sets the new entry's version
/// to parent.version + 1. Entries persisted before Phase 27 get
/// version=1 via serde default and are treated as roots.
#[serde(default = "default_version")]
pub version: u32,
/// Phase 27 — playbook_id of the prior version in this chain. None
/// for root entries (first version).
#[serde(default)]
pub parent_id: Option<String>,
/// Phase 27 — timestamp set when a newer version replaced this
/// entry via `revise_entry`. Superseded entries are excluded from
/// boost calculations (same rule as `retired_at`) but remain
/// queryable via `history` for audit.
#[serde(default)]
pub superseded_at: Option<String>,
/// Phase 27 — playbook_id of the entry that replaced this one.
/// Walking `superseded_by` from the root forward reconstructs the
/// full version chain.
#[serde(default)]
pub superseded_by: Option<String>,
}
fn default_version() -> u32 { 1 }
/// A recorded failure — worker who didn't deliver on a contract.
/// Tracked per (city, state, name) so a single worker's failures on
/// Toledo Welder contracts don't penalize the same name in Chicago.
@@ -168,6 +191,17 @@ pub enum UpsertOutcome {
Noop(String),
}
/// Phase 27 — shape returned from `revise_entry`. Reports both ends of
/// the supersession so callers can link citations or audit chains.
#[derive(Debug, Clone, Serialize)]
pub struct ReviseOutcome {
pub parent_id: String,
pub parent_version: u32,
pub new_playbook_id: String,
pub new_version: u32,
pub superseded_at: String,
}
/// Return YYYY-MM-DD from an RFC3339 timestamp. Falls back to the
/// first 10 chars if parse fails — tolerant for legacy entries that
/// stored a bare date.
@@ -204,13 +238,15 @@ impl PlaybookMemory {
/// Rebuild the geo index from scratch. Called by every mutation
/// helper after persist succeeds. O(n) scan of entries; at current
/// scale ~40µs. Skips retired entries — they never participate in
/// boost filtering, so indexing them would just waste lookups.
/// scale ~40µs. Skips retired and superseded entries — they never
/// participate in boost filtering, so indexing them would just
/// waste lookups.
async fn rebuild_geo_index(&self) {
let state = self.state.read().await;
let mut idx: HashMap<(String, String), Vec<usize>> = HashMap::new();
for (i, e) in state.entries.iter().enumerate() {
if e.retired_at.is_some() { continue; }
if e.superseded_at.is_some() { continue; }
let (Some(city), Some(st)) = (&e.city, &e.state) else { continue; };
let key = (city.to_ascii_lowercase(), st.to_ascii_uppercase());
idx.entry(key).or_default().push(i);
@@ -312,13 +348,129 @@ impl PlaybookMemory {
Ok(count)
}
/// Stats accessor for the /status endpoint and tests.
pub async fn status_counts(&self) -> (usize, usize, usize) {
/// Phase 27 — append a new version of an existing playbook. The
/// parent is stamped with `superseded_at` + `superseded_by`; the
/// new entry inherits `parent_id` and gets `version = parent + 1`.
/// Errors when the parent is retired (terminal state) or already
/// superseded (must revise the tip of the chain, not a middle
/// node). Caller supplies the new entry with its own fresh
/// `playbook_id`; chain-metadata fields on the input are
/// overwritten so callers can't fabricate a mismatched history.
pub async fn revise_entry(
&self,
parent_id: &str,
mut new_entry: PlaybookEntry,
) -> Result<ReviseOutcome, String> {
let now = chrono::Utc::now().to_rfc3339();
let mut state = self.state.write().await;
let Some(i) = state.entries.iter().position(|e| e.playbook_id == parent_id) else {
return Err(format!("parent playbook_id '{parent_id}' not found"));
};
{
let parent = &state.entries[i];
if parent.retired_at.is_some() {
return Err(format!(
"cannot revise retired playbook '{parent_id}' — retirement is terminal"
));
}
if let Some(succ) = &parent.superseded_by {
return Err(format!(
"playbook '{parent_id}' already superseded by '{succ}'; \
revise the latest version in the chain instead"
));
}
}
let parent_version = state.entries[i].version;
let new_version = parent_version.saturating_add(1);
let parent_pid = state.entries[i].playbook_id.clone();
let new_pid = new_entry.playbook_id.clone();
if new_pid.is_empty() {
return Err("new playbook_id must not be empty".into());
}
if new_pid == parent_pid {
return Err("new playbook_id must differ from parent".into());
}
// Enforce chain-metadata integrity — caller doesn't get to
// fabricate these.
new_entry.version = new_version;
new_entry.parent_id = Some(parent_pid.clone());
new_entry.superseded_at = None;
new_entry.superseded_by = None;
let parent_mut = &mut state.entries[i];
parent_mut.superseded_at = Some(now.clone());
parent_mut.superseded_by = Some(new_pid.clone());
state.entries.push(new_entry);
drop(state);
self.persist().await?;
self.rebuild_geo_index().await;
Ok(ReviseOutcome {
parent_id: parent_pid,
parent_version,
new_playbook_id: new_pid,
new_version,
superseded_at: now,
})
}
/// Phase 27 — return the full version chain that contains this
/// playbook_id, ordered from root (v1) to tip. Walks `parent_id`
/// backward to find the root, then `superseded_by` forward to the
/// tip. Returns empty if the id isn't present. Cycle-safe via a
/// visited set; unreachable in normal operation but the guard is
/// cheap.
pub async fn history(&self, playbook_id: &str) -> Vec<PlaybookEntry> {
let state = self.state.read().await;
let by_id: HashMap<&str, &PlaybookEntry> = state.entries
.iter()
.map(|e| (e.playbook_id.as_str(), e))
.collect();
let Some(seed) = by_id.get(playbook_id).copied() else {
return vec![];
};
// Walk backward to root.
let mut cursor = seed;
let mut seen: std::collections::HashSet<String> = std::collections::HashSet::new();
seen.insert(cursor.playbook_id.clone());
while let Some(pid) = &cursor.parent_id {
let Some(&next) = by_id.get(pid.as_str()) else { break };
if !seen.insert(next.playbook_id.clone()) { break; }
cursor = next;
}
let root = cursor;
// Walk forward to tip.
let mut chain = vec![root.clone()];
let mut cursor = root;
let mut seen_fwd: std::collections::HashSet<String> = std::collections::HashSet::new();
seen_fwd.insert(cursor.playbook_id.clone());
while let Some(nid) = &cursor.superseded_by {
let Some(&next) = by_id.get(nid.as_str()) else { break };
if !seen_fwd.insert(next.playbook_id.clone()) { break; }
cursor = next;
chain.push(cursor.clone());
}
chain
}
/// Stats accessor for the /status endpoint and tests. Returns
/// (total, retired, superseded, failures). Phase 27 added
/// superseded as a distinct counter: a superseded entry is
/// replaced-by-newer-version, which is a different lifecycle event
/// than retired-stop-using.
pub async fn status_counts(&self) -> (usize, usize, usize, usize) {
let state = self.state.read().await;
let total = state.entries.len();
let retired = state.entries.iter().filter(|e| e.retired_at.is_some()).count();
let superseded = state.entries.iter().filter(|e| e.superseded_at.is_some()).count();
let failures = state.failures.len();
(total, retired, failures)
(total, retired, superseded, failures)
}
/// Phase 26 — Mem0-style upsert. Decides ADD / UPDATE / NOOP based
@@ -353,6 +505,7 @@ impl PlaybookMemory {
let mut existing_idx: Option<usize> = None;
for (i, e) in state.entries.iter().enumerate() {
if e.retired_at.is_some() { continue; }
if e.superseded_at.is_some() { continue; }
if e.operation != new_entry.operation { continue; }
if day_key(&e.timestamp) != new_day { continue; }
if e.city != new_entry.city || e.state != new_entry.state { continue; }
@@ -514,6 +667,7 @@ impl PlaybookMemory {
.iter()
.filter(|e| {
if e.retired_at.is_some() { return false; }
if e.superseded_at.is_some() { return false; }
if let Some(vu) = &e.valid_until {
if let Ok(parsed) = chrono::DateTime::parse_from_rfc3339(vu) {
if now > parsed.with_timezone(&chrono::Utc) { return false; }
@@ -543,6 +697,7 @@ impl PlaybookMemory {
.filter_map(|i| state.entries.get(i))
.filter(|e| {
if e.retired_at.is_some() { return false; }
if e.superseded_at.is_some() { return false; }
if let Some(vu) = &e.valid_until {
if let Ok(parsed) = chrono::DateTime::parse_from_rfc3339(vu) {
if now > parsed.with_timezone(&chrono::Utc) { return false; }
@@ -1055,6 +1210,10 @@ pub async fn rebuild(
valid_until: None,
retired_at: None,
retirement_reason: None,
version: 1,
parent_id: None,
superseded_at: None,
superseded_by: None,
}
})
.collect();
@@ -1237,6 +1396,10 @@ mod tests {
valid_until: None,
retired_at: None,
retirement_reason: None,
version: 1,
parent_id: None,
superseded_at: None,
superseded_by: None,
})
.collect();
tokio::runtime::Runtime::new().unwrap().block_on(async {
@@ -1270,6 +1433,10 @@ mod validity_window_tests {
valid_until,
retired_at: None,
retirement_reason: None,
version: 1,
parent_id: None,
superseded_at: None,
superseded_by: None,
}
}
@@ -1279,7 +1446,7 @@ mod validity_window_tests {
pm.set_entries(vec![mkentry("pb-1", "Nashville", "TN", None, None)]).await.unwrap();
let touched = pm.retire_one("pb-1", "manual test").await.unwrap();
assert!(touched);
let (total, retired, _) = pm.status_counts().await;
let (total, retired, _, _) = pm.status_counts().await;
assert_eq!(total, 1);
assert_eq!(retired, 1);
// Second retirement is a no-op
@@ -1335,7 +1502,7 @@ mod validity_window_tests {
// Only pb-old-schema should be retired — pb-new-schema matches,
// pb-no-fp has no fingerprint so it's legacy-safe.
assert_eq!(retired, 1);
let (_, total_retired, _) = pm.status_counts().await;
let (_, total_retired, _, _) = pm.status_counts().await;
assert_eq!(total_retired, 1);
}
@@ -1348,7 +1515,7 @@ mod validity_window_tests {
// Nashville migration shouldn't touch Chicago
let retired = pm.retire_on_schema_drift("Nashville", "TN", "fp-v2", "test").await.unwrap();
assert_eq!(retired, 1);
let (_, r, _) = pm.status_counts().await;
let (_, r, _, _) = pm.status_counts().await;
assert_eq!(r, 1);
}
}
@@ -1373,6 +1540,10 @@ mod upsert_tests {
valid_until: None,
retired_at: None,
retirement_reason: None,
version: 1,
parent_id: None,
superseded_at: None,
superseded_by: None,
}
}
@@ -1444,3 +1615,171 @@ mod upsert_tests {
assert_eq!(pm.entry_count().await, 2);
}
}
#[cfg(test)]
mod version_tests {
use super::*;
use object_store::memory::InMemory;
fn mk(id: &str, city: &str, state: &str) -> PlaybookEntry {
PlaybookEntry {
playbook_id: id.into(),
operation: format!("fill: Welder x1 in {city}, {state}"),
approach: "hybrid".into(),
context: "test".into(),
timestamp: chrono::Utc::now().to_rfc3339(),
endorsed_names: vec!["Alice Smith".into()],
city: Some(city.into()),
state: Some(state.into()),
embedding: Some(vec![1.0, 0.0, 0.0]),
schema_fingerprint: None,
valid_until: None,
retired_at: None,
retirement_reason: None,
version: 1,
parent_id: None,
superseded_at: None,
superseded_by: None,
}
}
#[tokio::test]
async fn revise_stamps_chain_metadata_on_both_ends() {
let pm = PlaybookMemory::new(Arc::new(InMemory::new()));
pm.set_entries(vec![mk("pb-v1", "Nashville", "TN")]).await.unwrap();
let outcome = pm.revise_entry("pb-v1", mk("pb-v2", "Nashville", "TN"))
.await
.expect("revise should succeed against active root");
assert_eq!(outcome.parent_id, "pb-v1");
assert_eq!(outcome.parent_version, 1);
assert_eq!(outcome.new_playbook_id, "pb-v2");
assert_eq!(outcome.new_version, 2);
assert!(!outcome.superseded_at.is_empty());
let snap = pm.snapshot().await;
let v1 = snap.iter().find(|e| e.playbook_id == "pb-v1").unwrap();
let v2 = snap.iter().find(|e| e.playbook_id == "pb-v2").unwrap();
assert_eq!(v1.superseded_by.as_deref(), Some("pb-v2"));
assert!(v1.superseded_at.is_some());
assert_eq!(v2.parent_id.as_deref(), Some("pb-v1"));
assert_eq!(v2.version, 2);
assert!(v2.superseded_at.is_none());
}
#[tokio::test]
async fn revise_rejects_retired_parent() {
let pm = PlaybookMemory::new(Arc::new(InMemory::new()));
let mut e1 = mk("pb-v1", "Nashville", "TN");
e1.retired_at = Some(chrono::Utc::now().to_rfc3339());
pm.set_entries(vec![e1]).await.unwrap();
let err = pm.revise_entry("pb-v1", mk("pb-v2", "Nashville", "TN")).await
.expect_err("revise on retired parent must error");
assert!(err.contains("retired"), "error should mention retirement: {err}");
}
#[tokio::test]
async fn revise_rejects_already_superseded_parent() {
let pm = PlaybookMemory::new(Arc::new(InMemory::new()));
pm.set_entries(vec![mk("pb-v1", "Nashville", "TN")]).await.unwrap();
pm.revise_entry("pb-v1", mk("pb-v2", "Nashville", "TN")).await.unwrap();
// pb-v1 is now superseded; revising it again must fail — caller
// should revise pb-v2 (the tip) instead.
let err = pm.revise_entry("pb-v1", mk("pb-v3-fake", "Nashville", "TN")).await
.expect_err("revise on superseded parent must error");
assert!(err.contains("superseded"), "error should mention supersession: {err}");
}
#[tokio::test]
async fn superseded_entries_excluded_from_boost() {
let pm = PlaybookMemory::new(Arc::new(InMemory::new()));
pm.set_entries(vec![mk("pb-v1", "Nashville", "TN")]).await.unwrap();
let mut v2 = mk("pb-v2", "Nashville", "TN");
v2.endorsed_names = vec!["Carol Davis".into()];
pm.revise_entry("pb-v1", v2).await.unwrap();
let boosts = pm.compute_boost_for_filtered_with_role(
&[1.0, 0.0, 0.0], 100, 0.5,
Some(("Nashville", "TN")), Some("Welder"),
).await;
// v1's endorsement (Alice Smith) should be absent — it was
// superseded. v2's endorsement (Carol Davis) should be present.
assert!(
!boosts.contains_key(&("Nashville".into(), "TN".into(), "Alice Smith".into())),
"superseded entry's endorsement must not boost"
);
let carol = boosts.get(&("Nashville".into(), "TN".into(), "Carol Davis".into()));
assert!(carol.is_some(), "tip version's endorsement must still boost");
assert!(carol.unwrap().citations.contains(&"pb-v2".to_string()));
}
#[tokio::test]
async fn history_walks_root_to_tip_from_any_node() {
let pm = PlaybookMemory::new(Arc::new(InMemory::new()));
pm.set_entries(vec![mk("pb-v1", "Nashville", "TN")]).await.unwrap();
pm.revise_entry("pb-v1", mk("pb-v2", "Nashville", "TN")).await.unwrap();
pm.revise_entry("pb-v2", mk("pb-v3", "Nashville", "TN")).await.unwrap();
// Starting from the root — same chain.
let chain_from_root = pm.history("pb-v1").await;
assert_eq!(chain_from_root.len(), 3);
assert_eq!(chain_from_root[0].playbook_id, "pb-v1");
assert_eq!(chain_from_root[1].playbook_id, "pb-v2");
assert_eq!(chain_from_root[2].playbook_id, "pb-v3");
// Starting from the tip — same chain, same order.
let chain_from_tip = pm.history("pb-v3").await;
assert_eq!(chain_from_tip.len(), 3);
assert_eq!(chain_from_tip[0].playbook_id, "pb-v1");
assert_eq!(chain_from_tip[2].playbook_id, "pb-v3");
// Starting from the middle — same chain.
let chain_from_mid = pm.history("pb-v2").await;
assert_eq!(chain_from_mid.len(), 3);
assert_eq!(chain_from_mid[0].playbook_id, "pb-v1");
assert_eq!(chain_from_mid[2].playbook_id, "pb-v3");
}
#[tokio::test]
async fn history_empty_for_unknown_id() {
let pm = PlaybookMemory::new(Arc::new(InMemory::new()));
pm.set_entries(vec![mk("pb-v1", "Nashville", "TN")]).await.unwrap();
assert!(pm.history("pb-nonexistent").await.is_empty());
}
#[tokio::test]
async fn status_counts_reports_superseded_separately() {
let pm = PlaybookMemory::new(Arc::new(InMemory::new()));
pm.set_entries(vec![mk("pb-v1", "Nashville", "TN")]).await.unwrap();
pm.revise_entry("pb-v1", mk("pb-v2", "Nashville", "TN")).await.unwrap();
let (total, retired, superseded, _) = pm.status_counts().await;
assert_eq!(total, 2);
assert_eq!(retired, 0);
assert_eq!(superseded, 1);
}
#[tokio::test]
async fn legacy_entries_without_version_default_to_v1() {
// Simulate state persisted before Phase 27 — no version field.
// Serde default kicks in; entries should be treated as roots.
let json = r#"{
"entries": [{
"playbook_id": "pb-legacy",
"operation": "fill: Welder x1 in Nashville, TN",
"approach": "hybrid",
"context": "",
"timestamp": "2026-04-21T00:00:00Z",
"endorsed_names": ["Alice"],
"city": "Nashville",
"state": "TN"
}],
"last_rebuilt_at": 0,
"failures": []
}"#;
let state: PlaybookMemoryState = serde_json::from_str(json).unwrap();
let legacy = &state.entries[0];
assert_eq!(legacy.version, 1);
assert!(legacy.parent_id.is_none());
assert!(legacy.superseded_at.is_none());
assert!(legacy.superseded_by.is_none());
}
}