profit/golangLAKEHOUSE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

distillation: full SFT export port — closes OPEN #2 fully

Browse Source 
Follow-up to b216b7e (which shipped the SFT export substrate). This
commit ports the synthesis logic, completing the migration:

- SynthesizeSft(scored, ev, recordedAt, sftID) → *SftSample
  Mirrors the Rust synthesizeSft byte-for-byte. Returns nil for
  extraction-class records + empty-text records (same skip
  semantics as Rust).
- LoadEvidenceByRunID(scoredPath, cache) reads the paired evidence
  JSONL (path derived by /scored-runs/ → /evidence/ replacement).
  Per-call cache so multiple scored-runs files in the same dir
  don't reload the same evidence.
- buildInstruction maps source_file stem → per-class instruction
  template. All 8 templates (scrum_reviews, mode_experiments,
  auto_apply, audits, observer_reviews, contract_analyses,
  outcomes, default) match Rust output exactly so a/b validation
  between runtimes can diff JSONL byte-for-byte.
- stemFromSourceFile strips data/_kb/ prefix + .jsonl suffix.
- ExportSft now writes data/distilled/sft/sft_export.jsonl with
  the synthesized samples (DryRun=true skips file write).

Per-class templates verified by 8-case sub-test:
- scrum_reviews → "Review the file '...' against the PRD..."
- mode_experiments → "Run task_class='...' for file..."
- auto_apply → "Auto-apply: emit a 6-line surgical patch..."
- audits with phase: prefix → strips to bare phase name
- observer_reviews → "Observer-review the latest attempt..."
- contract_analyses with permit: prefix → strips to permit ID
- outcomes → "Run scenario; report per-event outcome..."
- unknown source → "Source 'X' run; produce the appropriate output"

Caveat documented inline: contract_analyses uses ev.metadata.contractor
in Rust to produce "Analyze contractor 'X' for permit 'Y'" when
present. Go's EvidenceRecord doesn't carry a free-form metadata bag
yet, so we always emit the no-contractor form. Operators needing
contractor-aware instructions can extend EvidenceRecord with an
explicit Metadata field (separate ADR).

Test additions (5 new):
- TestSynthesizeSft_PerSourceClass: 8 sub-cases, one per template
- TestSynthesizeSft_RejectsExtraction: extraction-role records skipped
- TestSynthesizeSft_RejectsEmptyText: empty/whitespace text skipped
- TestSynthesizeSft_ContextAssembly: matrix + pathway + model
  context string formatting matches Rust " · " join
- TestExportSft_FullPort_WritesJSONL: end-to-end fixture, asserts
  output contains expected instruction + omits firewalled records

Pre-existing TestExportSft_PartialPort_FirewallFires renamed +
updated to TestExportSft_FirewallFiresBeforeEvidenceLoad — reflects
the new contract that records passing the firewall but lacking
evidence land in "not-instructable" rather than being silently
exported. Honest semantics shift documented in the test.

OPEN #2 now fully closed (was: substrate-only). The synthesis path
no longer requires the Rust pipeline to be invoked — Go-side
operators can run the full distillation export end-to-end.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-05-01 00:06:57 -05:00
co-authored by Claude Opus 4.7
parent b216b7e5b6
commit 7bb432f6c8
3 changed files with 434 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/distillation/sft_export_test.go
+225 -14
View File
@@ -3,6 +3,7 @@ package distillation
import (
"os"
"path/filepath"
"strings"
"testing"
)
@@ -134,26 +135,233 @@ func TestListScoredRunFiles_WalksYearMonthDay(t *testing.T) {
}
}
// TestExportSft_PartialPort_FirewallFires runs the partial-port
// ExportSft on a fixture with one valid + one rejected ScoredRun
// and asserts the firewall counts correctly. Locks the contamination
// guarantee at the integration layer — even before the synthesis
// half ports, the firewall protection is end-to-end testable.
func TestExportSft_PartialPort_FirewallFires(t *testing.T) {
// TestSynthesizeSft_PerSourceClass locks the per-source-class
// instruction templates byte-for-byte against the Rust source.
// If a future commit changes a template, this test fails — the
// trained-model behavior shifts under our feet.
func TestSynthesizeSft_PerSourceClass(t *testing.T) {
cases := []struct {
name string
sourceFile string
taskID string
sourceFiles []string
wantPrefix string
}{
{
"scrum_reviews",
"data/_kb/scrum_reviews.jsonl",
"any",
[]string{"src/foo.rs"},
"Review the file 'src/foo.rs' against",
},
{
"mode_experiments",
"data/_kb/mode_experiments.jsonl",
"task_42",
[]string{"src/bar.go"},
"Run task_class='task_42' for file 'src/bar.go'.",
},
{
"auto_apply",
"data/_kb/auto_apply.jsonl",
"any",
[]string{"src/baz.ts"},
"Auto-apply: emit a 6-line surgical patch for 'src/baz.ts'",
},
{
"audits with phase: prefix stripped",
"data/_kb/audits.jsonl",
"phase:G2",
nil,
"Audit phase 'G2' and report findings",
},
{
"observer_reviews",
"data/_kb/observer_reviews.jsonl",
"any",
[]string{"f.rs"},
"Observer-review the latest attempt on 'f.rs'.",
},
{
"contract_analyses with permit: prefix",
"data/_kb/contract_analyses.jsonl",
"permit:ABC123",
nil,
"Analyze permit 'ABC123'. Recommend with risk markers.",
},
{
"outcomes",
"data/_kb/outcomes.jsonl",
"any",
nil,
"Run scenario; report per-event outcome with citations.",
},
{
"unknown source falls back to default",
"data/_kb/something_new.jsonl",
"any",
nil,
"Source 'something_new' run; produce the appropriate output",
},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
scored := ScoredRun{
EvidenceRunID: "rid",
Category: CategoryAccepted,
Provenance: Provenance{
SourceFile: c.sourceFile,
SigHash: "abc",
RecordedAt: "2026-04-30T00:00:00Z",
},
}
ev := EvidenceRecord{
RunID: "rid",
TaskID: c.taskID,
ModelRole: RoleExecutor,
Text: "model response text",
SourceFiles: c.sourceFiles,
Provenance: Provenance{SourceFile: c.sourceFile, SigHash: "abc", RecordedAt: "2026-04-30T00:00:00Z"},
}
sample := SynthesizeSft(scored, ev, "2026-04-30T00:00:00Z", "test-id")
if sample == nil {
t.Fatalf("expected non-nil sample for %s", c.name)
}
if !strings.HasPrefix(sample.Instruction, c.wantPrefix) {
t.Errorf("instruction prefix mismatch:\n got: %q\n want: %q...", sample.Instruction, c.wantPrefix)
}
})
}
}
// TestSynthesizeSft_RejectsExtraction: extraction-class records
// have no instruction→response shape (they're pure data extraction,
// not model-output-as-training-target). Synthesis must return nil.
func TestSynthesizeSft_RejectsExtraction(t *testing.T) {
ev := EvidenceRecord{
RunID: "rid",
ModelRole: RoleExtractor,
Text: "extracted data",
Provenance: Provenance{SourceFile: "data/_kb/anything.jsonl", SigHash: "abc", RecordedAt: "2026-04-30T00:00:00Z"},
}
scored := ScoredRun{EvidenceRunID: "rid", Category: CategoryAccepted, Provenance: ev.Provenance}
if sample := SynthesizeSft(scored, ev, "2026-04-30T00:00:00Z", "id"); sample != nil {
t.Errorf("extraction record must produce nil sample, got %+v", sample)
}
}
// TestSynthesizeSft_RejectsEmptyText: text is the response side of
// the SFT pair; empty text means nothing to learn.
func TestSynthesizeSft_RejectsEmptyText(t *testing.T) {
ev := EvidenceRecord{
RunID: "rid",
ModelRole: RoleExecutor,
Text: " \n\t",
Provenance: Provenance{SourceFile: "data/_kb/scrum_reviews.jsonl", SigHash: "abc", RecordedAt: "2026-04-30T00:00:00Z"},
}
scored := ScoredRun{EvidenceRunID: "rid", Category: CategoryAccepted, Provenance: ev.Provenance}
if sample := SynthesizeSft(scored, ev, "2026-04-30T00:00:00Z", "id"); sample != nil {
t.Errorf("empty-text record must produce nil sample, got %+v", sample)
}
}
// TestSynthesizeSft_ContextAssembly verifies the terse " · "-joined
// context string carries matrix corpora + pathway fingerprints +
// model name in the documented order.
func TestSynthesizeSft_ContextAssembly(t *testing.T) {
ev := EvidenceRecord{
RunID: "rid",
ModelRole: RoleReviewer,
Text: "verdict",
ModelName: "qwen3.5",
RetrievedContext: &RetrievedContext{
MatrixCorpora: []string{"workers", "candidates"},
PathwayFingerprintsSeen: 88,
},
Provenance: Provenance{SourceFile: "data/_kb/scrum_reviews.jsonl", SigHash: "abc", RecordedAt: "2026-04-30T00:00:00Z"},
}
scored := ScoredRun{EvidenceRunID: "rid", Category: CategoryAccepted, Provenance: ev.Provenance}
sample := SynthesizeSft(scored, ev, "2026-04-30T00:00:00Z", "id")
if sample == nil {
t.Fatalf("expected non-nil sample")
}
want := "matrix=workers,candidates · pathway_fingerprints=88 · model=qwen3.5"
if sample.Context != want {
t.Errorf("context mismatch:\n got: %q\n want: %q", sample.Context, want)
}
}
// TestExportSft_FullPort_WritesJSONL covers the fully-ported path:
// scored runs + paired evidence both present, synthesis produces
// SftSamples, output JSONL is written. Locks the end-to-end
// contract that next-wave changes (synthesis tweaks, output layout)
// have to preserve.
func TestExportSft_FullPort_WritesJSONL(t *testing.T) {
tmp := t.TempDir()
scoredDir := filepath.Join(tmp, "data", "scored-runs", "2026", "04", "30")
evidenceDir := filepath.Join(tmp, "data", "evidence", "2026", "04", "30")
for _, d := range []string{scoredDir, evidenceDir} {
if err := os.MkdirAll(d, 0o755); err != nil {
t.Fatalf("mkdir %s: %v", d, err)
}
}
scoredJSONL := `{"category":"accepted","evidence_run_id":"r1","provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"h1","recorded_at":"2026-04-30T00:00:00Z"}}
{"category":"rejected","evidence_run_id":"r2","provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"h2","recorded_at":"2026-04-30T00:00:00Z"}}
`
evidenceJSONL := `{"run_id":"r1","model_role":"executor","text":"some review output","source_files":["src/foo.rs"],"provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"h1","recorded_at":"2026-04-30T00:00:00Z"}}
{"run_id":"r2","model_role":"executor","text":"another output","source_files":["src/bar.rs"],"provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"h2","recorded_at":"2026-04-30T00:00:00Z"}}
`
if err := os.WriteFile(filepath.Join(scoredDir, "run.jsonl"), []byte(scoredJSONL), 0o644); err != nil {
t.Fatalf("write scored: %v", err)
}
if err := os.WriteFile(filepath.Join(evidenceDir, "run.jsonl"), []byte(evidenceJSONL), 0o644); err != nil {
t.Fatalf("write evidence: %v", err)
}
res, err := ExportSft(ExportSftOptions{
Root: tmp,
RecordedAt: "2026-04-30T00:00:00Z",
})
if err != nil {
t.Fatalf("ExportSft: %v", err)
}
if res.RecordsRead != 2 || res.RecordsExported != 1 || res.RecordsQuarantined != 1 {
t.Errorf("counts: read=%d exported=%d quarantined=%d (want 2/1/1)",
res.RecordsRead, res.RecordsExported, res.RecordsQuarantined)
}
out, err := os.ReadFile(res.OutputPath)
if err != nil {
t.Fatalf("read output: %v", err)
}
if !strings.Contains(string(out), "Review the file 'src/foo.rs'") {
t.Errorf("output missing expected scrum_reviews instruction; got:\n%s", string(out))
}
if strings.Contains(string(out), "src/bar.rs") {
t.Errorf("output contains rejected record's source_file — firewall leak")
}
}
// TestExportSft_FirewallFiresBeforeEvidenceLoad locks the order-of-
// operations: even if evidence records are missing, the
// firewall counts records as quarantined so the contamination
// guarantee never depends on side data being present. Records
// that pass the firewall but lack evidence get the more honest
// "not instructable" label rather than being silently exported.
func TestExportSft_FirewallFiresBeforeEvidenceLoad(t *testing.T) {
tmp := t.TempDir()
dir := filepath.Join(tmp, "data", "scored-runs", "2026", "04", "30")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatalf("mkdir: %v", err)
}
// Two scored runs: one passes the firewall, one is blocked.
jsonl := `{"category":"accepted","run_id":"r1","task_id":"t1"}
{"category":"rejected","run_id":"r2","task_id":"t2"}
{"category":"partially_accepted","run_id":"r3","task_id":"t3"}
{"category":"needs_human_review","run_id":"r4","task_id":"t4"}
jsonl := `{"category":"accepted","evidence_run_id":"r1","provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"h1","recorded_at":"2026-04-30T00:00:00Z"}}
{"category":"rejected","evidence_run_id":"r2","provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"h2","recorded_at":"2026-04-30T00:00:00Z"}}
{"category":"partially_accepted","evidence_run_id":"r3","provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"h3","recorded_at":"2026-04-30T00:00:00Z"}}
{"category":"needs_human_review","evidence_run_id":"r4","provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"h4","recorded_at":"2026-04-30T00:00:00Z"}}
`
if err := os.WriteFile(filepath.Join(dir, "run.jsonl"), []byte(jsonl), 0o644); err != nil {
t.Fatalf("write: %v", err)
}
// No evidence directory created → records that pass the firewall
// land in "not instructable" since synthesis can't proceed.
res, err := ExportSft(ExportSftOptions{
Root: tmp,
RecordedAt: "2026-04-30T00:00:00Z",
@@ -165,10 +373,13 @@ func TestExportSft_PartialPort_FirewallFires(t *testing.T) {
if res.RecordsRead != 4 {
t.Errorf("RecordsRead: got %d, want 4", res.RecordsRead)
}
if res.RecordsExported != 2 {
t.Errorf("RecordsExported (firewall-passing): got %d, want 2", res.RecordsExported)
}
if res.RecordsQuarantined != 2 {
t.Errorf("RecordsQuarantined (firewall-blocked): got %d, want 2", res.RecordsQuarantined)
}
if res.RecordsExported != 0 {
t.Errorf("RecordsExported with no evidence: got %d, want 0", res.RecordsExported)
}
if !strings.Contains(res.QuarantineSummary, "not-instructable=2") {
t.Errorf("expected quarantine summary to flag 2 not-instructable, got %q", res.QuarantineSummary)
}
}