profit/golangLAKEHOUSE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

playbook_lift v2: paraphrase pass + run #002 finds boost-only limit

Browse Source 
Adds an opt-in Pass 3 to the lift driver: for each query whose Pass 1
recorded a playbook, ask the judge to rephrase the query, then re-query
with playbook=true and check whether the recorded answer surfaces in
top-K. This is the test the v1 report's caveat #3 explicitly flagged
as the actual learning-property gate (not the cheap verbatim case).

Implementation:
- New flag --with-paraphrase on the driver (default off).
- New WITH_PARAPHRASE env in the harness (default 1, on for prod runs).
- New paraphrase_* fields on queryRun + summary, // 0 fallback in jq so
  re-rendering verbatim-only evidence stays clean.
- generateParaphrase() calls the same judge model with format=json and
  a tight schema; temperature=0.5 for variance without domain drift.
- Markdown report adds a paraphrase per-query table (only when the
  pass ran) and an honesty caveat about judge-also-rephrases coupling.

Run #002 result (reports/reality-tests/playbook_lift_002.{json,md}):

  Verbatim lift               2/2 (100% — Q7 + Q13, both stable from v1)
  Paraphrase top-1            0/2
  Paraphrase any-rank in K    0/2

Both paraphrases dropped the recorded answer OUT of top-K entirely
(rank=-1). This isn't a paraphrase-quality problem — qwen2.5's outputs
preserved intent ("Hazmat-certified warehouse worker comfortable with
cold storage" → "Warehouse worker with Hazmat certification and
experience in cold storage"). It's the v0 boost-only stance documented
in internal/matrix/playbook.go:22-27: the boost only re-ranks results
that ALREADY surfaced from regular retrieval. If paraphrase's cosine
retrieval doesn't include the recorded answer in top-K, no boost can
promote it.

The "Shape B" upgrade mentioned in the playbook.go comment — inject
playbook hits directly even when they weren't in the top-K — is what
would close this gap. The reality test surfaced exactly the gap the
docs warned about. Worth filing as the next product gate.

Run-to-run variance also visible: v1 had 8 discoveries, v2 had 2.
HNSW insertion order + judge variance both contribute. Stability of
Q7 and Q13 across both runs (lifted in v1 AND v2) is the most reliable
signal in the dataset.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-30 06:47:41 -05:00
co-authored by Claude Opus 4.7
parent 9ce067bd9d
commit e9822f025d
3 changed files with 322 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/playbook_lift/main.go
+139 -3
View File
@@ -81,6 +81,15 @@ type queryRun struct {
WarmJudgeBestRank int `json:"warm_judge_best_rank"`
Lift bool `json:"lift"` // judge-best was below top-1 cold, but top-1 warm
// Paraphrase pass — only populated when --with-paraphrase. Tests
// the playbook's actual learning property: does a recorded entry
// for query Q help a similar-but-different query Q'?
ParaphraseQuery string `json:"paraphrase_query,omitempty"`
ParaphraseTop1ID string `json:"paraphrase_top1_id,omitempty"`
ParaphraseRecordedRank int `json:"paraphrase_recorded_rank,omitempty"` // -1 if not in top-K
ParaphraseLift bool `json:"paraphrase_lift,omitempty"` // recorded answer at rank 0 for paraphrase
Note string `json:"note,omitempty"`
}
@@ -91,7 +100,13 @@ type summary struct {
NoChange int `json:"no_change"`
MeanTop1DeltaDistance float32 `json:"mean_top1_delta_distance"`
PlaybookBoostedTotal int `json:"playbook_boosted_total"`
GeneratedAt time.Time `json:"generated_at"`
// Paraphrase pass aggregates — only populated when --with-paraphrase.
ParaphraseAttempted int `json:"paraphrase_attempted,omitempty"` // queries with playbook recorded that ran a paraphrase
ParaphraseTop1Lifts int `json:"paraphrase_top1_lifts,omitempty"` // recorded answer surfaced at rank 0
ParaphraseAnyRankHits int `json:"paraphrase_any_rank_hits,omitempty"` // recorded answer surfaced at any rank in top-K
GeneratedAt time.Time `json:"generated_at"`
}
func main() {
@@ -104,6 +119,7 @@ func main() {
judge := flag.String("judge", "", "Ollama model for relevance judging (empty = read from config [models].local_judge)")
k := flag.Int("k", 10, "top-k from matrix.search per pass")
out := flag.String("out", "reports/reality-tests/playbook_lift_001.json", "output JSONL path")
withParaphrase := flag.Bool("with-paraphrase", false, "after warm pass, generate a paraphrase via the judge model and re-query with playbook=true to test the learning property")
flag.Parse()
// Judge resolution priority: explicit flag > $JUDGE_MODEL env >
@@ -226,6 +242,59 @@ func main() {
totalDelta += runs[i].WarmTop1Distance - runs[i].ColdTop1Distance
}
// Pass 3 (paraphrase) — opt-in via --with-paraphrase. For each
// query where a playbook was recorded in Pass 1, generate a
// paraphrase via the judge model and run it through warm
// matrix.search. The expectation: if the playbook's learning
// property holds (cosine on embed(paraphrase) finds the recorded
// embed(query) within DefaultPlaybookMaxDistance), the recorded
// answer should appear at top-1 for the paraphrase too. This is
// the claim from the report's caveat #3 that v1 didn't test.
paraphraseAttempted := 0
paraphraseTop1Lifts := 0
paraphraseAnyRankHits := 0
if *withParaphrase {
log.Printf("[lift] paraphrase pass: testing playbook learning property")
for i := range runs {
if !runs[i].PlaybookRecorded {
continue
}
paraphraseAttempted++
paraphrase, err := generateParaphrase(hc, *ollama, *judge, runs[i].Query)
if err != nil {
log.Printf(" (%d) paraphrase generation failed: %v", i+1, err)
runs[i].Note = appendNote(runs[i].Note, "paraphrase gen failed: "+err.Error())
continue
}
runs[i].ParaphraseQuery = paraphrase
log.Printf("[lift] (%d/%d paraphrase) %s → %s", i+1, len(runs),
abbrev(runs[i].Query, 40), abbrev(paraphrase, 40))
resp, err := matrixSearch(hc, *gw, paraphrase, corpora, *k, true)
if err != nil || len(resp.Results) == 0 {
runs[i].Note = appendNote(runs[i].Note, fmt.Sprintf("paraphrase search failed: %v", err))
runs[i].ParaphraseRecordedRank = -1
continue
}
runs[i].ParaphraseTop1ID = resp.Results[0].ID
recordedRank := -1
for j, r := range resp.Results {
if r.ID == runs[i].PlaybookID {
recordedRank = j
break
}
}
runs[i].ParaphraseRecordedRank = recordedRank
if recordedRank == 0 {
runs[i].ParaphraseLift = true
paraphraseTop1Lifts++
paraphraseAnyRankHits++
} else if recordedRank > 0 {
paraphraseAnyRankHits++
}
}
}
sum := summary{
Total: len(runs),
WithDiscovery: withDiscovery,
@@ -233,6 +302,9 @@ func main() {
NoChange: noChange,
MeanTop1DeltaDistance: 0,
PlaybookBoostedTotal: playbookBoostedTotal,
ParaphraseAttempted: paraphraseAttempted,
ParaphraseTop1Lifts: paraphraseTop1Lifts,
ParaphraseAnyRankHits: paraphraseAnyRankHits,
GeneratedAt: time.Now().UTC(),
}
if len(runs) > 0 {
@@ -242,11 +314,75 @@ func main() {
if err := writeJSON(*out, runs, sum); err != nil {
log.Fatalf("write %s: %v", *out, err)
}
log.Printf("[lift] DONE — %d queries · discovery=%d · lift=%d · boosted=%d · meanΔdist=%.4f",
sum.Total, sum.WithDiscovery, sum.LiftCount, sum.PlaybookBoostedTotal, sum.MeanTop1DeltaDistance)
if *withParaphrase {
log.Printf("[lift] DONE — %d queries · discovery=%d · lift=%d · boosted=%d · meanΔdist=%.4f · paraphrase=%d/%d→top1, %d/%d→anyrank",
sum.Total, sum.WithDiscovery, sum.LiftCount, sum.PlaybookBoostedTotal, sum.MeanTop1DeltaDistance,
sum.ParaphraseTop1Lifts, sum.ParaphraseAttempted,
sum.ParaphraseAnyRankHits, sum.ParaphraseAttempted)
} else {
log.Printf("[lift] DONE — %d queries · discovery=%d · lift=%d · boosted=%d · meanΔdist=%.4f",
sum.Total, sum.WithDiscovery, sum.LiftCount, sum.PlaybookBoostedTotal, sum.MeanTop1DeltaDistance)
}
log.Printf("[lift] results → %s", *out)
}
// generateParaphrase asks the judge model to rephrase a staffing query
// while preserving intent. Used in the paraphrase pass to test whether
// the playbook's recorded embedding survives wording variation.
//
// temperature=0.5 — enough variance to make the paraphrase actually
// different, but not so high that it drifts off the staffing domain.
// format=json + a tight schema makes parsing deterministic.
func generateParaphrase(hc *http.Client, ollamaURL, model, query string) (string, error) {
system := `You rephrase staffing queries while preserving intent.
Output JSON only: {"paraphrase": "<rephrased query>"}.
Rules:
- Keep the same role, certifications, geography, and constraints.
- Vary the wording (synonyms, reordered clauses, different sentence shape).
- Do NOT add or remove requirements.
- Do NOT explain — just emit the JSON.`
body := map[string]any{
"model": model,
"stream": false,
"format": "json",
"messages": []map[string]string{
{"role": "system", "content": system},
{"role": "user", "content": query},
},
"options": map[string]any{"temperature": 0.5},
}
bs, _ := json.Marshal(body)
req, _ := http.NewRequest("POST", ollamaURL+"/api/chat", bytes.NewReader(bs))
req.Header.Set("Content-Type", "application/json")
resp, err := hc.Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode/100 != 2 {
return "", fmt.Errorf("ollama chat: HTTP %d", resp.StatusCode)
}
rb, _ := io.ReadAll(resp.Body)
var ollamaResp struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
}
if err := json.Unmarshal(rb, &ollamaResp); err != nil {
return "", fmt.Errorf("decode ollama envelope: %w", err)
}
var out struct {
Paraphrase string `json:"paraphrase"`
}
if err := json.Unmarshal([]byte(ollamaResp.Message.Content), &out); err != nil {
return "", fmt.Errorf("decode paraphrase JSON: %w (content=%q)", err, ollamaResp.Message.Content)
}
if strings.TrimSpace(out.Paraphrase) == "" {
return "", fmt.Errorf("empty paraphrase (content=%q)", ollamaResp.Message.Content)
}
return out.Paraphrase, nil
}
func loadQueries(path string) ([]string, error) {
bs, err := os.ReadFile(path)
if err != nil {