playbook_lift v2: paraphrase pass + run #002 finds boost-only limit
Adds an opt-in Pass 3 to the lift driver: for each query whose Pass 1 recorded a playbook, ask the judge to rephrase the query, then re-query with playbook=true and check whether the recorded answer surfaces in top-K. This is the test the v1 report's caveat #3 explicitly flagged as the actual learning-property gate (not the cheap verbatim case). Implementation: - New flag --with-paraphrase on the driver (default off). - New WITH_PARAPHRASE env in the harness (default 1, on for prod runs). - New paraphrase_* fields on queryRun + summary, // 0 fallback in jq so re-rendering verbatim-only evidence stays clean. - generateParaphrase() calls the same judge model with format=json and a tight schema; temperature=0.5 for variance without domain drift. - Markdown report adds a paraphrase per-query table (only when the pass ran) and an honesty caveat about judge-also-rephrases coupling. Run #002 result (reports/reality-tests/playbook_lift_002.{json,md}): Verbatim lift 2/2 (100% — Q7 + Q13, both stable from v1) Paraphrase top-1 0/2 Paraphrase any-rank in K 0/2 Both paraphrases dropped the recorded answer OUT of top-K entirely (rank=-1). This isn't a paraphrase-quality problem — qwen2.5's outputs preserved intent ("Hazmat-certified warehouse worker comfortable with cold storage" → "Warehouse worker with Hazmat certification and experience in cold storage"). It's the v0 boost-only stance documented in internal/matrix/playbook.go:22-27: the boost only re-ranks results that ALREADY surfaced from regular retrieval. If paraphrase's cosine retrieval doesn't include the recorded answer in top-K, no boost can promote it. The "Shape B" upgrade mentioned in the playbook.go comment — inject playbook hits directly even when they weren't in the top-K — is what would close this gap. The reality test surfaced exactly the gap the docs warned about. Worth filing as the next product gate. Run-to-run variance also visible: v1 had 8 discoveries, v2 had 2. HNSW insertion order + judge variance both contribute. Stability of Q7 and Q13 across both runs (lifted in v1 AND v2) is the most reliable signal in the dataset. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
@@ -46,6 +46,12 @@ QUERIES_FILE="${QUERIES_FILE:-tests/reality/playbook_lift_queries.txt}"
|
||||
CORPORA="${CORPORA:-workers,ethereal_workers}"
|
||||
K="${K:-10}"
|
||||
CONFIG_PATH="${CONFIG_PATH:-lakehouse.toml}"
|
||||
# WITH_PARAPHRASE=1 (default) adds a Pass 3 — for each query whose
|
||||
# Pass 1 cold pass recorded a playbook, generate a paraphrase via the
|
||||
# judge and re-query with playbook=true. The paraphrase pass is the
|
||||
# actual learning-property test (does cosine on paraphrase find the
|
||||
# recorded entry?). Set WITH_PARAPHRASE=0 for a faster verbatim-only run.
|
||||
WITH_PARAPHRASE="${WITH_PARAPHRASE:-1}"
|
||||
|
||||
OUT_JSON="reports/reality-tests/playbook_lift_${RUN_ID}.json"
|
||||
OUT_MD="reports/reality-tests/playbook_lift_${RUN_ID}.md"
|
||||
@@ -265,6 +271,10 @@ echo "[lift] running driver — judge=$EFFECTIVE_JUDGE · queries=$QUERIES_FILE
|
||||
# and runs its own resolution chain (env → config → fallback). When
|
||||
# JUDGE_MODEL IS set, the explicit -judge wins inside the Go driver
|
||||
# regardless of what its env-lookup would find — flag wins by design.
|
||||
PARAPHRASE_FLAG=""
|
||||
if [ "$WITH_PARAPHRASE" = "1" ]; then
|
||||
PARAPHRASE_FLAG="-with-paraphrase"
|
||||
fi
|
||||
./bin/playbook_lift \
|
||||
-config "$CONFIG_PATH" \
|
||||
-gateway "http://127.0.0.1:3110" \
|
||||
@@ -273,13 +283,15 @@ echo "[lift] running driver — judge=$EFFECTIVE_JUDGE · queries=$QUERIES_FILE
|
||||
-corpora "$CORPORA" \
|
||||
-judge "$JUDGE_MODEL" \
|
||||
-k "$K" \
|
||||
-out "$OUT_JSON"
|
||||
-out "$OUT_JSON" \
|
||||
$PARAPHRASE_FLAG
|
||||
|
||||
echo
|
||||
echo "[lift] generating markdown report → $OUT_MD"
|
||||
generate_md() {
|
||||
local json="$1" md="$2"
|
||||
local total discovery lift no_change boosted mean_delta gen_at
|
||||
local p_attempted p_top1 p_anyrank p_block
|
||||
total=$(jq -r '.summary.total' "$json")
|
||||
discovery=$(jq -r '.summary.with_discovery' "$json")
|
||||
lift=$(jq -r '.summary.lift_count' "$json")
|
||||
@@ -287,6 +299,18 @@ generate_md() {
|
||||
boosted=$(jq -r '.summary.playbook_boosted_total' "$json")
|
||||
mean_delta=$(jq -r '.summary.mean_top1_delta_distance' "$json")
|
||||
gen_at=$(jq -r '.summary.generated_at' "$json")
|
||||
p_attempted=$(jq -r '.summary.paraphrase_attempted // 0' "$json")
|
||||
p_top1=$(jq -r '.summary.paraphrase_top1_lifts // 0' "$json")
|
||||
p_anyrank=$(jq -r '.summary.paraphrase_any_rank_hits // 0' "$json")
|
||||
|
||||
# Only emit the paraphrase block when --with-paraphrase actually ran
|
||||
# (i.e. .summary.paraphrase_attempted > 0). For verbatim-only runs we
|
||||
# leave the headline clean.
|
||||
p_block=""
|
||||
if [ "$p_attempted" != "0" ] && [ "$p_attempted" != "null" ]; then
|
||||
p_block="| **Paraphrase pass — recorded answer at rank 0 (top-1)** | **${p_top1} / ${p_attempted}** |
|
||||
| Paraphrase pass — recorded answer at any rank in top-K | ${p_anyrank} / ${p_attempted} |"
|
||||
fi
|
||||
|
||||
cat > "$md" <<MDEOF
|
||||
# Playbook-Lift Reality Test — Run ${RUN_ID}
|
||||
@@ -297,6 +321,7 @@ generate_md() {
|
||||
**Workers limit:** ${WORKERS_LIMIT}
|
||||
**Queries:** \`${QUERIES_FILE}\` (${total} executed)
|
||||
**K per pass:** ${K}
|
||||
**Paraphrase pass:** $([ "$WITH_PARAPHRASE" = "1" ] && echo "ENABLED" || echo "disabled")
|
||||
**Evidence:** \`${OUT_JSON}\`
|
||||
|
||||
---
|
||||
@@ -311,8 +336,9 @@ generate_md() {
|
||||
| No change (judge-best already top-1, no playbook needed) | ${no_change} |
|
||||
| Playbook boosts triggered (warm pass) | ${boosted} |
|
||||
| Mean Δ top-1 distance (warm − cold) | ${mean_delta} |
|
||||
${p_block}
|
||||
|
||||
**Lift rate:** ${lift} of ${discovery} discoveries became top-1 after warm pass.
|
||||
**Verbatim lift rate:** ${lift} of ${discovery} discoveries became top-1 after warm pass.
|
||||
|
||||
---
|
||||
|
||||
@@ -335,6 +361,39 @@ MDEOF
|
||||
] | "| " + join(" | ") + " |"
|
||||
' "$json" >> "$md"
|
||||
|
||||
# Paraphrase per-query table — only emit when the pass ran, and only
|
||||
# for queries where Pass 1 recorded a playbook (others have no
|
||||
# paraphrase_query field).
|
||||
if [ "$p_attempted" != "0" ] && [ "$p_attempted" != "null" ]; then
|
||||
cat >> "$md" <<MDEOF
|
||||
|
||||
---
|
||||
|
||||
## Paraphrase pass — does the playbook help similar-but-different queries?
|
||||
|
||||
For each query whose Pass 1 cold pass recorded a playbook entry, the
|
||||
judge model rephrased the query, and the rephrased version was sent
|
||||
through warm matrix.search. The recorded answer ID's rank in those
|
||||
results tests whether cosine on the embedded paraphrase finds the
|
||||
recorded query's vector.
|
||||
|
||||
| # | Original (≤40c) | Paraphrase (≤60c) | Recorded answer | Paraphrase top-1 | Recorded rank | Paraphrase lift |
|
||||
|---|---|---|---|---|---|---|
|
||||
MDEOF
|
||||
jq -r '.runs | to_entries[] |
|
||||
select(.value.playbook_recorded == true and (.value.paraphrase_query // "") != "") |
|
||||
[
|
||||
(.key + 1 | tostring),
|
||||
(.value.query | .[0:40]),
|
||||
((.value.paraphrase_query // "") | .[0:60]),
|
||||
(.value.playbook_target_id // "—"),
|
||||
(.value.paraphrase_top1_id // "—"),
|
||||
(.value.paraphrase_recorded_rank | tostring),
|
||||
(if .value.paraphrase_lift then "**YES**" else "no" end)
|
||||
] | "| " + join(" | ") + " |"
|
||||
' "$json" >> "$md"
|
||||
fi
|
||||
|
||||
cat >> "$md" <<MDEOF
|
||||
|
||||
---
|
||||
@@ -349,15 +408,23 @@ MDEOF
|
||||
\`distance' = distance × (1 - 0.5 × score)\`. Lift requires the judge-best
|
||||
result's pre-boost distance to be ≤ 2× the cold top-1's distance, otherwise
|
||||
even halving doesn't promote it. Tight clusters → little visible lift.
|
||||
3. **Same-query replay is the cheap case.** Real lift comes from *similar but
|
||||
not identical* queries hitting a recorded playbook. This run only tests
|
||||
verbatim replay. A v2 should add paraphrase queries.
|
||||
3. **Verbatim vs paraphrase.** The verbatim lift rate (above) is the cheap
|
||||
case — same query, recorded playbook, expected boost. The paraphrase
|
||||
pass (when enabled) is the actual learning property: similar-but-different
|
||||
queries hitting a recorded playbook. Compare verbatim and paraphrase
|
||||
lift rates — paraphrase should be lower (semantic-distance gates some
|
||||
playbook hits) but non-zero is the meaningful signal.
|
||||
4. **Multi-corpus skew.** Default corpora=\`${CORPORA}\` — if all judge-best
|
||||
results land in one corpus, the matrix layer's purpose isn't being tested.
|
||||
Check per-corpus distribution in the JSON.
|
||||
5. **Judge resolution.** This run used \`${EFFECTIVE_JUDGE}\` from
|
||||
${JUDGE_SOURCE}.
|
||||
Bumping the judge for run #N+1 means editing one line in lakehouse.toml.
|
||||
6. **Paraphrase generation also uses the judge.** The same model that rates
|
||||
relevance also rephrases queries. A judge that's bad at rating staffing
|
||||
queries is probably also bad at rephrasing them. Worth sanity-checking
|
||||
a sample of \`paraphrase_query\` values in the JSON before trusting the
|
||||
paraphrase lift number.
|
||||
|
||||
## Next moves
|
||||
|
||||
|
||||
Reference in New Issue
Block a user