profit/golangLAKEHOUSE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

multi-coord stress Phase 1.5: shared-role contracts + paraphrase handover

Browse Source 
Phase 1 had two known gaps: (1) the 3 contracts had zero shared role
names, so same-role-across-contracts Jaccard was vacuous (n=0); (2)
the verbatim handover at 100% was the trivial case, not the hard
learning test (paraphrased queries against another coord's playbook).

Both fixed in this commit.

Contract redesign — all 3 contracts now share warehouse worker /
admin assistant / heavy equipment operator roles, plus a unique
specialist per contract (industrial electrician / bilingual safety
coord / drone surveyor — the "specialist not on the standard roster"
case from J's spec). Counts and skill mixes vary per region.

New driver phase 4b — paraphrase handover. Bob runs qwen2.5-paraphrased
versions of Alice's contract queries against Alice's playbook
namespace. Tests whether institutional memory propagates across
coordinators AND across natural wording variation that Bob would
introduce when running Alice's contract.

Run #002 result (5K workers + 10K ethereal_workers, 4 demand × 3
coords + paraphrase handover):

  Diversity (the question J asked: locking or cycling?):
    Same-role-across-contracts Jaccard = 0.119 (n=9)
      → 88% of workers DIFFER across regions for the same role name.
        Milwaukee warehouse vs Indianapolis warehouse vs Chicago
        warehouse pull mostly distinct top-K from the same population.
        The system locks into geo+cert+skill context, not cycling.
    Different-roles-same-contract Jaccard = 0.004 (n=18)
      → role-specific retrieval works (unchanged from Phase 1).

  Determinism: Jaccard = 1.000 (n=12) — unchanged.

  Learning:
    Verbatim handover  4/4 = 100%  (trivial case, expected)
    Paraphrase handover 4/4 = 100% (HARD case — passes!)
      Of those 4 paraphrase recoveries:
        - 2 used boost (Alice's recording was already in Bob's
          paraphrase top-K; ApplyPlaybookBoost re-ranked to top-1)
        - 2 used Shape B inject (recording wasn't in Bob's
          paraphrase top-K; InjectPlaybookMisses brought it in)

The boost/inject mix is healthy — both paths are used and both
produce correct top-1s. Multi-coord institutional memory propagation
is empirically working under wording variation.

Sample warehouse worker top-1s across contracts (proves diversity):
  alice / Milwaukee     → w-713
  bob   / Indianapolis  → e-8447
  carol / Chicago       → e-7145
Three different workers from the same 15K-person population,
selected on geo+cert+skill context.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-30 08:03:16 -05:00
co-authored by Claude Opus 4.7
parent 61c7b55e48
commit 0fa42a0cc3
5 changed files with 240 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/multi_coord_stress.sh
+23 -7
View File
@@ -155,12 +155,19 @@ echo "[stress] ingest ethereal_workers (limit=$ETHEREAL_LIMIT, 0=all) into 'ethe
echo
echo "[stress] running multi-coord stress driver..."
EXTRA_FLAGS=""
if [ "${WITH_PARAPHRASE_HANDOVER:-1}" = "1" ]; then
EXTRA_FLAGS="$EXTRA_FLAGS -with-paraphrase-handover"
fi
./bin/multi_coord_stress \
-gateway "http://127.0.0.1:3110" \
-contracts tests/reality/contracts \
-corpora "$CORPORA" \
-k "$K" \
-out "$OUT_JSON"
-out "$OUT_JSON" \
-ollama "http://localhost:11434" \
-judge "${JUDGE_MODEL:-qwen2.5:latest}" \
$EXTRA_FLAGS
echo
echo "[stress] generating markdown report → $OUT_MD"
@@ -179,6 +186,10 @@ hand_run=$(jq -r '.learning.handover_queries_run' "$OUT_JSON")
hand_top1=$(jq -r '.learning.recorded_answers_top1_count' "$OUT_JSON")
hand_topk=$(jq -r '.learning.recorded_answers_topk_count' "$OUT_JSON")
hand_rate=$(jq -r '.learning.handover_hit_rate' "$OUT_JSON")
ph_run=$(jq -r '.learning.paraphrase_handover_run // 0' "$OUT_JSON")
ph_top1=$(jq -r '.learning.paraphrase_top1_count // 0' "$OUT_JSON")
ph_topk=$(jq -r '.learning.paraphrase_topk_count // 0' "$OUT_JSON")
ph_rate=$(jq -r '.learning.paraphrase_handover_hit_rate // 0' "$OUT_JSON")
cat > "$OUT_MD" <<MDEOF
# Multi-Coordinator Stress Test — Run ${RUN_ID}
@@ -227,14 +238,19 @@ Bob takes Alice's contract using Alice's playbook namespace. Did Alice's recorde
| Metric | Value |
|---|---:|
| Handover queries run | ${hand_run} |
| Alice's recorded answer at Bob's top-1 | ${hand_top1} |
| Alice's recorded answer in Bob's top-K | ${hand_topk} |
| **Handover hit rate (top-1)** | **${hand_rate}** |
| Verbatim handover queries run | ${hand_run} |
| Alice's recorded answer at Bob's top-1 (verbatim) | ${hand_top1} |
| Alice's recorded answer in Bob's top-K (verbatim) | ${hand_topk} |
| **Verbatim handover hit rate (top-1)** | **${hand_rate}** |
| Paraphrase handover queries run | ${ph_run} |
| Alice's recorded answer at Bob's top-1 (paraphrase) | ${ph_top1} |
| Alice's recorded answer in Bob's top-K (paraphrase) | ${ph_topk} |
| **Paraphrase handover hit rate (top-1)** | **${ph_rate}** |
**Interpretation:**
- Hit rate ≥ 0.5: handover is meaningful — the second coordinator inherits the first's institutional memory.
- Hit rate ≈ 0.0: playbook namespace isolation is working but the playbook itself isn't transferable, OR Bob's queries don't match Alice's recordings closely enough.
- Verbatim hit rate ≈ 1.0: trivial case — Bob runs identical queries; should always hit.
- Paraphrase hit rate ≥ 0.5: institutional memory survives wording change — the harder learning property.
- Paraphrase hit rate ≈ 0.0: Bob's paraphrases drift past the inject threshold, so Alice's recordings don't activate. Same caveat as the playbook_lift paraphrase pass.
---