Scanner respects .gitignore — full Lakehouse Rust scan now possible
Single biggest unblock for using the harness on real targets. The lakehouse Rust repo has a 67GB data/ directory holding parquet, JSONL pathway memory, headshots, and other runtime data — all gitignored. Pre-fix the scanner walked it all (and stalled). Post- fix the full Rust scan completes in 15s. internal/scanner/gitignore.go — minimal Matcher that handles the patterns real .gitignore files use ~99% of the time: - basename match anywhere (`pattern`) - dir-only match (`pattern/`) - root-anchored (`/pattern`) - path-anchored (`pattern/sub` — interior slash) - extension globs (`*.ext`) - path + extension (`path/*.ext`) - comments + blank lines ignored Negations (!pattern) intentionally NOT supported v0; matcher records HasNegations() so callers can surface a warning if encountered. internal/scanner/gitignore_test.go — 14 cases against a synthetic .gitignore covering all 6 pattern shapes, plus missing-file and negation-recording tests. walk.go integration: gitignore loaded once at scan start; checked in the dir-skip branch (SkipDir cascades) and the file-emit branch. Skip layers in order: universal-noise basenames → .gitignore → path-scoped self-skip → dotfile filter. Verified end-to-end: - lakehouse Rust full repo: 15s scan, 1031 findings, 0 critical (no committed secrets in source — independently confirms what scrum2 + the Rust auditor said) - 529 hardcoded-path findings IS the Sprint 4 gap the audit kept naming; the harness just put a number on it This was Opus's WARN B5 from the cross-lineage scrum, plus the "harness stalls on real repos" gap exposed when running it against the actual Lakehouse repos. Both addressed in one wave. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
ab550a7c5a
commit
2fc047487f
@@ -81,6 +81,16 @@ type Result struct {
|
||||
// Walk produces a Result for repoPath. Errors on missing dir; skipped
|
||||
// dirs are silently filtered. countLines is true → reads each file
|
||||
// for line counts (Phase B needs this; Phase A wires false for speed).
|
||||
//
|
||||
// Skip layers, applied in order:
|
||||
// 1. Universal-noise basenames (.git, node_modules, vendor, etc.)
|
||||
// 2. .gitignore patterns (read once at repoPath/.gitignore)
|
||||
// 3. Path-scoped self-skip (only when the harness scans its own tree)
|
||||
// 4. Dotfile filter (skip leading-dot files unless interestingDotfile)
|
||||
//
|
||||
// Layer 2 is the "scrum fix B5+1" — without it, target repos with
|
||||
// large data dirs (lakehouse Rust has 67GB under data/) stalled the
|
||||
// scan. With it, the scanner respects the operator's intent.
|
||||
func Walk(repoPath string, countLines bool) (*Result, error) {
|
||||
abs, err := filepath.Abs(repoPath)
|
||||
if err != nil {
|
||||
@@ -104,6 +114,12 @@ func Walk(repoPath string, countLines bool) (*Result, error) {
|
||||
// docstring for B5 background.
|
||||
pathSkips := selfSkipsFor(abs)
|
||||
|
||||
// Load .gitignore at repo root if present. Empty/missing →
|
||||
// matcher.Skip always returns false; behavior matches the pre-
|
||||
// 2026-04-30 scanner. Negations (!pattern) are NOT supported v0;
|
||||
// matcher.HasNegations() lets the caller surface a warning.
|
||||
gitignore, _ := LoadGitignore(abs)
|
||||
|
||||
walkErr := filepath.WalkDir(abs, func(p string, d fs.DirEntry, walkErr error) error {
|
||||
if walkErr != nil {
|
||||
return nil // best-effort; permission errors etc. are silent
|
||||
@@ -115,6 +131,12 @@ func Walk(repoPath string, countLines bool) (*Result, error) {
|
||||
if pathSkips[p] {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
if gitignore.Skip(p, true) {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if gitignore.Skip(p, false) {
|
||||
return nil
|
||||
}
|
||||
// Skip dotfiles at file level (.gitignore etc. are interesting,
|
||||
|
||||
Reference in New Issue
Block a user