Scanner respects .gitignore — full Lakehouse Rust scan now possible

Single biggest unblock for using the harness on real targets. The
lakehouse Rust repo has a 67GB data/ directory holding parquet,
JSONL pathway memory, headshots, and other runtime data — all
gitignored. Pre-fix the scanner walked it all (and stalled). Post-
fix the full Rust scan completes in 15s.

internal/scanner/gitignore.go — minimal Matcher that handles the
patterns real .gitignore files use ~99% of the time:
  - basename match anywhere (`pattern`)
  - dir-only match (`pattern/`)
  - root-anchored (`/pattern`)
  - path-anchored (`pattern/sub` — interior slash)
  - extension globs (`*.ext`)
  - path + extension (`path/*.ext`)
  - comments + blank lines ignored

Negations (!pattern) intentionally NOT supported v0; matcher records
HasNegations() so callers can surface a warning if encountered.

internal/scanner/gitignore_test.go — 14 cases against a synthetic
.gitignore covering all 6 pattern shapes, plus missing-file and
negation-recording tests.

walk.go integration: gitignore loaded once at scan start; checked
in the dir-skip branch (SkipDir cascades) and the file-emit branch.
Skip layers in order: universal-noise basenames → .gitignore →
path-scoped self-skip → dotfile filter.

Verified end-to-end:
- lakehouse Rust full repo: 15s scan, 1031 findings, 0 critical
  (no committed secrets in source — independently confirms what
  scrum2 + the Rust auditor said)
- 529 hardcoded-path findings IS the Sprint 4 gap the audit kept
  naming; the harness just put a number on it

This was Opus's WARN B5 from the cross-lineage scrum, plus the
"harness stalls on real repos" gap exposed when running it against
the actual Lakehouse repos. Both addressed in one wave.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude (review-harness setup)
2026-04-30 02:14:10 -05:00
co-authored by Claude Opus 4.7
parent ab550a7c5a
commit 2fc047487f
3 changed files with 320 additions and 0 deletions
+22
View File
@@ -81,6 +81,16 @@ type Result struct {
// Walk produces a Result for repoPath. Errors on missing dir; skipped
// dirs are silently filtered. countLines is true → reads each file
// for line counts (Phase B needs this; Phase A wires false for speed).
//
// Skip layers, applied in order:
// 1. Universal-noise basenames (.git, node_modules, vendor, etc.)
// 2. .gitignore patterns (read once at repoPath/.gitignore)
// 3. Path-scoped self-skip (only when the harness scans its own tree)
// 4. Dotfile filter (skip leading-dot files unless interestingDotfile)
//
// Layer 2 is the "scrum fix B5+1" — without it, target repos with
// large data dirs (lakehouse Rust has 67GB under data/) stalled the
// scan. With it, the scanner respects the operator's intent.
func Walk(repoPath string, countLines bool) (*Result, error) {
abs, err := filepath.Abs(repoPath)
if err != nil {
@@ -104,6 +114,12 @@ func Walk(repoPath string, countLines bool) (*Result, error) {
// docstring for B5 background.
pathSkips := selfSkipsFor(abs)
// Load .gitignore at repo root if present. Empty/missing →
// matcher.Skip always returns false; behavior matches the pre-
// 2026-04-30 scanner. Negations (!pattern) are NOT supported v0;
// matcher.HasNegations() lets the caller surface a warning.
gitignore, _ := LoadGitignore(abs)
walkErr := filepath.WalkDir(abs, func(p string, d fs.DirEntry, walkErr error) error {
if walkErr != nil {
return nil // best-effort; permission errors etc. are silent
@@ -115,6 +131,12 @@ func Walk(repoPath string, countLines bool) (*Result, error) {
if pathSkips[p] {
return filepath.SkipDir
}
if gitignore.Skip(p, true) {
return filepath.SkipDir
}
return nil
}
if gitignore.Skip(p, false) {
return nil
}
// Skip dotfiles at file level (.gitignore etc. are interesting,