Skip to content

feat(#15-#23): nine enhancements + diff-aware --since rewrite + safety caps - #24

Merged
machado144 merged 17 commits into
mainfrom
feat/issues-15-to-23-enhancements
Apr 8, 2026
Merged

machado144 merged 17 commits into
mainfrom
feat/issues-15-to-23-enhancements

Conversation

@machado144

Copy link
Copy Markdown
Contributor

Closes #15, #16, #17, #18, #19, #20, #21, #22, #23.

Summary

This branch implements all nine enhancement issues, then layers on a perf rewrite of --since, three safety knobs for unattended use (--max-files, --max-pairs, --quiet), reporter de-noising, and a hero demo GIF. 272 tests passing (started the branch at 217), go vet clean, lefthook (gofumpt + structlint + govet + golangci-lint + commitlint) green on every commit.

Issues addressed

# Feature Status
#15 ** glob in --exclude ✓
#16 skip binary files ✓
#17 --include allowlist ✓
#18 .dupehound-ignore + inline dupehound:ignore markers (all 20 languages) ✓
#19 --top N output verbosity ✓
#20 --since <ref> diff-aware scanning (now a real cost reducer) ✓
#21 --dead-code heuristic ✓
#22 test↔prod span detection ✓
#23 --git-churn ranking ✓

What's in the branch beyond the bare features

Reporter de-noising (fix(reporter))

  • Removed unconditional All clones: dump that was silently undoing --top
  • Stopped double-listing test↔prod clones (top section now iterates normalClones)
  • Capped clone instance lists to 6 with ... N more locations (was unbounded — clone with 28 instances dumped 28 lines on real repos)
  • Added 🧪 test↔prod / 🔥 churn / 🚫 suppressed badges to markdown clone summaries (PR comments previously lost churn entirely)
  • Capped dead-functions lists in both formats; markdown overflow wraps in <details>
  • Throttled fuzzy-detector progress logs from "every 10k pairs" to ~20 emissions max

--since perf rewrite (feat(detector))

  • --since was a post-detection filter — paid the full O(N²) fuzzy cost on every file. Threaded InScopeFiles []bool through DetectOptions so the exact and fuzzy detectors skip work for clones whose instances are all in unchanged files
  • Fixed a silent path-form bug: dupehound scan --path . --since main used to always report 0 clones (relative-vs-absolute path mismatch in the lookup map)
  • Measured on cli/cli (816 Go files): full scan 2:21 → --since HEAD~10 23s, ~6× faster, identical correctness, works with both relative and absolute --path

Safety knobs for unattended use (feat(safety))

  • --max-files N — fails fast in the file collector with ErrTooManyFiles and an actionable error message
  • --max-pairs N — aborts fuzzy detection (returning exact-only + a WARN log) before allocating the multi-GB pair dedup map
  • --quiet, -q — maps to zerolog warn level. INFO-level progress logs are silenced; the scan report on stdout is unaffected, so failing pre-commit hooks still print the full clone listing — exactly what a developer or LLM-based reviewer needs to act on the failure

Documentation

  • New EXAMPLES.md cookbook with 12 collapsible recipes, copy-pasteable, all anchors verified
  • README rewritten with TOC, refreshed flag list, deep-links into EXAMPLES.md
  • Hero demo GIF (docs/demo/dupehound-demo.gif, 188 KB) recorded against real axeforging/tacomex-8bit-shop showing dedup + dead-code in action. Spec is cast.yaml (declarative, idempotent — clones tacomex on first run, short-circuits after); regenerable with one command. Intermediates gitignored, only the final GIF is tracked.

Test additions (217 → 272, +55 meaningful tests)

Each new feature has regression, feature, and barrier (negative / edge / security) tests:

  • feat: support ** recursive glob patterns in --exclude #15-feat: --include flag to allowlist files/patterns for scanning #17: glob_test.go strict negative for --include — non-matching files absent from TotalFiles/ScannedFiles/FileStats, not just from clones
  • feat: clone suppression via .dupehound-ignore and inline annotations #18: parametrized inline-ignore across all 20 languages × 4 comment styles, plus end-to-end Python exclusion (different function-body marker than Go), plus over-eager-match negative cases
  • feat: --git-churn flag to rank clones by git change frequency #23: real-git-repo end-to-end churn test — builds a temp repo with known commit history (a.go ×4, b.go ×1), asserts per-instance FileCommits, total ChurnScore, and the descending sort invariant
  • --since detector scope: 8 tests covering exact path, fuzzy path, nil-preserves-behavior, out-of-range safety, real-repo end-to-end, no-changes-no-clones barrier, relative-path regression
  • Hard caps: 6 tests covering off-by-one boundary, MaxFiles=0 "no cap" guard, after-include-filter ordering, fuzzy abort barrier, MaxPairs=0 regression
  • --quiet: 2 end-to-end integration tests proving stderr is silenced AND the scan report on stdout is preserved (the failure-reason guarantee for hooks)
  • Security regression: TestResolveGitRef_DashPrefix and TestResolveGitRef_ValidSHA for the --since ref injection defense

Security review

No new attack surface introduced. Specifically:

  • New flags are pure integer comparisons + a log level mapping
  • --since path normalization uses filepath.Clean and filepath.Abs (deterministic stdlib) on paths the scanner already trusted
  • The git ref still flows through resolveGitRef which rejects dash-prefixed refs and converts to a hex SHA before any subsequent git invocation. Two-layer defense (dash-prefix rejection + SHA-only usage downstream)
  • One fewer subprocess invocation per scan because getChangedLines is now reused instead of called twice
  • go vet clean, all pre-commit hooks green

Known follow-ups (intentionally out of scope)

  • No timeout flag. Recommended timeout 10m dupehound … wrapper documented in EXAMPLES.md
  • --staged is still a post-filter (only --since was rewritten). Pair with --similarity 1.0 for cheap staged scans, or use --since HEAD --staged together
  • Hotspot/instance ordering nondeterminism — surfaced (and worked around) by the --quiet test. Sort tie-breaker isn't fully deterministic. Cosmetic but noticeable in run-to-run diffs

Test plan

  • All 272 existing + new unit tests pass: `go test ./...`
  • `go vet ./...` clean
  • `gofumpt -l ./...` clean
  • structlint clean
  • commitlint clean (conventional commit format on every commit)
  • Trial run on `cli/cli` (816 Go files) — verified text + md + churn + since + include modes produce useful, non-noisy output
  • Trial run on `tacomex-8bit-shop` — verified hero demo GIF reproduces correctly from a clean state
  • Reviewer: spot-check the EXAMPLES.md anchors render correctly on GitHub
  • Reviewer: confirm the hero GIF loads and is readable on GitHub README

🤖 Generated with Claude Code

Add new fields to Report, Clone, CloneInstance structs for binary skip,
suppression, test-prod span, dead code, diff-aware scanning, and churn.
Introduce --include, --top, --since, --show-suppressed, --dead-code,
--git-churn, --churn-days flags. Extend ScanConfig with include/top.
Add gobwas/glob for ** glob support. Add coverage.out to .gitignore
and .structlint.yaml ignore list.
Binary file skip (#16): before tokenizing, read first 512 bytes and skip
files containing a null byte; increment skipped_files in report.

Double-star glob in --exclude (#15): replace filepath.Match with gobwas/glob
for ** support. Patterns without ** continue to match basenames as before.

Include filter (#17): add --include flag using same ** glob support. If set,
only files matching at least one include pattern are scanned. Without --include
all files are scanned (backward compatible).

Add BuildTokenizedFileWithIgnore for inline dupehound:ignore annotation support.
Extend TokenizedFile with Ignored field. Add markIgnoredBlocks to tokenizer.
…ode, test-prod, churn

Clone suppression (#18): .dupehound-ignore file support with hash prefix,
path glob (** supported), and file:line range rules. Inline //dupehound:ignore
annotation excludes function bodies from detection. --show-suppressed flag.

Top-N output (#19): --top flag limits text/md output (0=all). JSON/SARIF
always return full output.

Diff-aware scanning (#20): --since <ref> filters clones to those touching
lines changed since the git ref. Adds new_clones to report JSON.

Dead code detection (#21): --dead-code flag identifies functions with no
external callers using token-based heuristic analysis.

Test↔prod clone annotation (#22): annotate clones spanning test and non-test
files with TestProdSpan=true and IsTest=true on instances. Shown as separate
section in text/md output. SARIF uses ruleId DUPE002 for these.

Git churn (#23): --git-churn re-sorts clones by commit activity (--churn-days
window, default 90 days). Annotates instances with FileCommits.
Scanner: invoke suppression, test-prod annotation, diff filtering, churn
scoring, and dead code detection in the scan pipeline. Binary files
increment skipped_files counter.

Reporter: add test-prod section in text/md output, --top-N truncation,
dead function report with heuristic disclaimer, since-diff header,
churn score/commit labels. SARIF uses DUPE002 for test-prod clones.

Scan action: wire all new CLI flags to ScanOptions and FormatOptions.
Exit code 1 when --since finds new clones or --dead-code finds dead
functions (--exit-zero still overrides). --min-duplication applies to
new-clone percentage when --since is active.
Binary skip: isBinaryData unit tests, scan skips binary files and
counts them in skipped_files.

Glob/include: matchGlob with **, matchesIncludes, scan with --exclude
vendor/**, --exclude **/*_test.go, --include **/*.go, combined
include+exclude.

Top-N: topNWithOpt unit test, text output truncated at --top 3,
--top -1 shows all, JSON unaffected.

Test-prod: isTestFile for all languages, annotateTestProdSpan mixed/
both-prod/both-test, full scan annotation.

Suppression: by hash, path glob, file:line range, --show-suppressed,
no ignore file, loading from actual file, parseIgnoreLine parsing.

Dead code: called function not dead, unused function detected, flag
absent means no output, JSON/text output format.

Diff filtering: parseDiffHunks, filterClonesInDiff, overlapsChangedLines,
invalid ref graceful failure.

Churn: sortByChurn ordering, tie-break, git-churn absent means default
sort, JSON/text output with churn fields.

Inline ignore: //dupehound:ignore marks Go tokens, # dupehound:ignore
marks Python tokens, isIgnoreComment variants.
- Reject --since refs starting with '-' or containing shell-unsafe
  characters (space, semicolon, backtick, dollar) to prevent git from
  interpreting user input as option flags (e.g. --output=, --upload-pack=)
- Add TestValidateGitRef covering valid refs (origin/main, HEAD~1, tags)
  and invalid inputs (flags, empty string, shell metacharacters)
- Fix matchGlob: remove redundant second glob.Compile that tested
  filepath.Base against a ** pattern — semantically wrong since
  vendor/** can never match "user.go", and compiled the same pattern twice
Replace character-allowlist validation with resolveGitRef() which calls
`git rev-parse --verify <ref>` and returns the full 40-char hex SHA.
All downstream git diff calls now use only the SHA — a hex string that
can never be misinterpreted as a flag regardless of what the user passes.

The '-' prefix guard is kept as a fast-path to reject obvious flag-shaped
inputs before forking a git process at all.

Add tests:
- TestResolveGitRef_DashPrefix: flag-shaped inputs rejected without git
- TestResolveGitRef_ValidSHA: HEAD resolves to a safe hex-only string
Tests added (+34, total 256 passing):
- #17 strict negative test: --include must exclude non-matching files from
  TotalFiles, ScannedFiles, FileStats — not just from clones
- #23 real-git-repo end-to-end churn test: builds a temp repo with known
  commit history (a.go x4, b.go x1) and asserts FileCommits + ChurnScore
  are correct, plus the churn-descending sort invariant
- #18 parametrized inline-ignore across all 20 supported languages and 4
  comment styles (//, #, --, /* */); negative cases for over-eager match;
  Python end-to-end exclusion test (different function-body marker than Go);
  strengthened TestMarkIgnoredBlocks_ExcludesFromDetection from a no-op
  TotalFiles assertion to a real cross-file exclusion check

Reporter (text + markdown) noise fixes:
- Remove unconditional "All clones:" dump that defeated --top in text format
- Stop double-listing test↔prod clones (top section now iterates normalClones)
- Cap test↔prod section by --top with overflow note
- Cap clone instance lists to 6 (was unbounded — clone with 28 instances
  dumped 28 lines); --verbose still shows all
- Add test↔prod / churn / suppressed badges to markdown clone summaries
  (PR comments previously lost churn entirely)
- Per-instance commit counts in markdown locations
- Cap dead-functions list in both formats; markdown overflow in details
- Fix markdown early-return on TotalClones==0 dropping dead functions

Detector log throttling (CI / hook UX):
- Aggregate fuzzy bucket truncated warnings into a single summary line
- Throttle fuzzy detection progress to ~20 emissions max regardless of
  pair count (was every 10k → thousands of lines/sec on fast machines)

Trial on cli/cli (816 Go files): output went from thousands of stderr
log lines to 24, and clone instance lists are now bounded.
EXAMPLES.md is a copy-pasteable cookbook organized in collapsible sections
for the features added in #15-#23. Topics: pre-commit hooks (lefthook +
plain git), GH Actions PR comments, diff-aware --since, include/exclude
globs, .dupehound-ignore + inline suppression, test↔prod leak detection,
git churn ranking, dead-function detection, safe-mode profiles for large
monorepos, output formats, and a real-world trial on cli/cli with measured
times and findings.

README.md adds a Contents TOC, refreshes the flag list with all the new
options (--include, --since, --top, --git-churn, --churn-days, --dead-code,
--show-suppressed), and adds a "More examples & use cases" section that
deep-links into EXAMPLES.md so the README itself stays terse.
The --since flag was a post-detection filter: it ran a full clone scan,
then dropped clones whose instances didn't touch the diff. On a 800-file
repo with a 30-file PR diff this paid the full O(N²) fuzzy detection cost
for ~3% of the files we cared about.

This commit threads an InScopeFiles []bool through DetectOptions and uses it
in two places:

- detectExact: when emitting a clone group, skip it (and mark its positions
  covered) if none of the participating files are in scope.
- detectFuzzy: when building candidate pairs, skip pairs where neither
  block is in a file marked in scope. This is the dominant cost win — it
  cuts both the pair count and the dedup map size.

The scanner resolves the in-scope file set BEFORE detection by calling
getChangedLines once and reusing the result for the existing line-precise
NewClones post-filter (one fewer git invocation as a side benefit).

Path normalization fix: getChangedLines returns absolute paths, but
tokenized files carry whatever form the walker produced. With --path .
the lookup never matched and --since silently produced 0 clones. The
scanner now rekeys the changedLines map by the same form as
tokenizedFiles[i].Path, so --since works correctly with both relative
and absolute --path.

Measured on cli/cli (816 Go files): full scan 2:21 → --since HEAD~10 23s,
same 211 in-scope clones, identical regardless of relative/absolute --path.

Tests added (8 total):

Unit (DetectWithOptions, no scanner):
- TestDetector_InScopeFilter_OnlyReportsClonesTouchingScope: a 3-file
  scenario where only one file is in scope; clone is emitted because at
  least one instance touches the scope set
- TestDetector_InScopeFilter_DropsAllOutOfScopeClones: empty scope ⇒ 0
  clones (the fast path that --since relies on)
- TestDetector_InScopeFilter_NilPreservesOriginalBehavior: regression
  guard against anyone making nil mean "nothing in scope"
- TestDetector_InScopeFilter_FuzzyPath: scope filter is honored on the
  type-3 path, not just the exact path
- TestDetector_InScopeFilter_OutOfRangeIndexIsSafe: defensive — short
  scope slice doesn't panic; out-of-range indices are treated as in scope

End-to-end (real git repo via Scanner.Scan):
- TestScanner_Since_RealRepo_FewerCloneEvaluations: builds a temp repo
  with known commit history, asserts --since produces ≤ as many clones as
  full scan AND the in-diff file is surfaced in NewClones
- TestScanner_Since_NoChangesNoClones: --since HEAD on a repo with no
  diff produces zero clones regardless of how much duplication exists
  (the safety guarantee that makes --since usable in pre-commit hooks)
- TestScanner_Since_RelativePath: regression for the silent path-form
  bug — runs --since with --path "." and asserts the diff is picked up
…d use

Three safety nets for running dupehound on huge monorepos and inside hooks
without OOM or stderr noise.

--max-files N (default 0 = no cap)
  Hard cap checked after include/exclude filtering. If the file walk
  collected more files than the cap, the scanner returns ErrTooManyFiles
  with an actionable message naming both the count and the cap. Catches
  misconfigured --path before any tokenization runs.

--max-pairs N (default 0 = no cap)
  Hard cap on fuzzy candidate pairs. The detector aborts type-3 detection
  (returning whatever exact clones it had) before the dedup map can grow
  past the cap. Surfaces a WARN-level log explaining the fix: lower
  --max-bucket, raise --max-pairs, or disable fuzzy with --similarity 1.0.
  Prevents the multi-GB pair dedup map on pathological repos.

--quiet, -q
  Maps to zerolog warn level. INFO-level progress logs (starting scan,
  scope summary, fuzzy progress) are suppressed; warnings, errors, and the
  scan report itself are unaffected. The report goes to stdout via
  fmt.Print, so a failing pre-commit hook still prints the full clone
  listing — file paths, line ranges, and code preview — exactly what a
  developer or LLM-based reviewer needs to fix the problem. --verbose
  wins over --quiet (more output is the safer default for ambiguous intent).

Tests added (8 total):

services/limits_test.go (6):
- TestScanner_MaxFiles_FailsFast: feature test — error wraps
  ErrTooManyFiles, message mentions actual numbers and remediation flags
- TestScanner_MaxFiles_Disabled: regression — MaxFiles=0 ⇒ no cap
- TestScanner_MaxFiles_AtBoundary: off-by-one barrier (= passes, +1 fails)
- TestScanner_MaxFiles_AppliedAfterIncludeFilter: cap is checked against
  the post-filter file count, not the raw walk
- TestDetector_MaxPairs_AbortsFuzzy: tight cap on a many-near-miss corpus
  produces a strict subset of the baseline and contains zero type-3 clones
- TestDetector_MaxPairs_Zero: regression — MaxPairs=0 ⇒ no cap

integration/cli_test.go (2):
- TestQuiet_SuppressesInfoLogsButKeepsReport: end-to-end — baseline scan
  emits "starting scan" on stderr, --quiet does not, but stdout still
  contains "Clones found" and "Top clones" (the failure-reason guarantee)
- TestQuiet_VerboseWinsOverQuiet: precedence regression — both flags ⇒
  verbose level

Security review of all changes in this commit and the preceding --since
commit: no new attack surface. The new flags are pure integer comparisons
and a log level mapping. The --since path normalization uses filepath.Clean
and filepath.Abs (deterministic stdlib functions) on paths the scanner
already trusted. The git ref still flows through resolveGitRef which
rejects dash-prefixed refs and converts to a hex SHA before any subsequent
git invocation. go vet clean.
EXAMPLES.md:
- Diff-aware section now reflects that --since is a true cost reducer,
  not a post-filter; includes the cli/cli measurement (2:21 → 23s)
- Safe-mode "big picture" rewritten with the full safety knob hierarchy:
  --similarity 1.0, --max-pairs, --max-files, --max-bucket, --since,
  --include — in order of impact
- Profile A (pre-commit) and Profile B (CI on PR diff) updated to add
  --quiet and to use --since for the CI profile
- New Profile D for paranoid unattended scans with all hard caps applied
- New "Hook output and AI readability" section explaining the stream
  contract: --quiet only filters info logs on stderr, the clone report
  on stdout (the failure reason) is unaffected, so failing hooks remain
  fully usable for developers and LLM-based reviewers
- Quick reference table and TOC link to all new sections

README.md:
- Flag list refreshed with --max-files, --max-pairs, --quiet
- --since description updated to mention it's a real cost reduction
- Deep-link list points to the new safe-mode and hook-output sections
Single hero demo at the top of EXAMPLES.md and README.md, recorded against
a real public sample app (axeforging/tacomex-8bit-shop — React + Fastify,
69 source files, ~14k LOC). Two acts:

1. Plain scan finds 163 clones / 20% duplication. Top hit is a 38-line
   type-2 block repeated 11 times in backend/src/seeds/index.ts (the taco
   data) — a clear extract-helper opportunity, not a fake fixture.
2. --dead-code surfaces real candidates: closeConnection, optionalAuth,
   requireAdmin — actual functions that look genuinely dead in the repo.

The demo is reproducible. docs/demo/ contains:

- cast.yaml: declarative spec for the recording (tracked)
- setup.sh: idempotent — clones tacomex into /tmp/dupehound-demo-repo (tracked)
- README.md: regeneration instructions (tracked)
- dupehound-demo.gif: 191 KB final render (tracked)

Intermediates that should not be committed are in .gitignore:

- docs/demo/*.cast (asciinema raw recording)
- docs/demo/.bin/ (any locally-built binary)

structlint: allow docs/** paths and *.gif / *.sh extensions; ignore the
.cast intermediate so it doesn't trip the structure validator after a
local re-record.

Why one hero instead of one-per-recipe: per-recipe GIFs go stale on every
flag/output change (we just shipped --quiet, --max-files, --max-pairs, and
the --since rewrite — every recipe recording from yesterday would already
need re-recording). The text recipes in EXAMPLES.md remain the canonical
copy-paste source; the hero GIF is the "what does this thing do" entry
point that benefits from being seen, not read.
Regenerating the GIF was a two-step dance: bash docs/demo/setup.sh, then
run cast. Move the (idempotent) clone into a `bash:` scene at the top of
cast.yaml so the whole thing is one command:

  python3 path/to/cast.py docs/demo/cast.yaml

The clone uses `test -d /tmp/dupehound-demo-repo || git clone …` so re-runs
short-circuit instantly. setup.sh deleted; structlint config no longer
needs the *.sh allowance. docs/demo/README.md updated with the new
single-command workflow.

Re-recorded from a clean state to verify the inlined clone path works.
GIF size unchanged (~188 KB), both acts (dedup + dead-code) intact.
@github-actions

github-actions Bot commented Apr 8, 2026 •

Copy link
Copy Markdown

StructLint — All checks passed

72 rules validated against .structlint.yaml. No violations found.

View full run · Powered by StructLint

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This Pull Request introduces a suite of significant enhancements to dupehound, focusing on improved usability, performance, and new detection capabilities. Key features include:

  • Advanced File Filtering: Implemented ** glob support for --exclude and a new --include allowlist, providing more granular control over scanned files.
  • Code Suppression: Added support for .dupehound-ignore files and inline //dupehound:ignore comments across all 20 supported languages, allowing users to suppress known or intentional duplicates.
  • Diff-Aware Scanning (--since): Rewrote the --since functionality to be a true performance optimization, where the detector actively skips work for clones not touching changed files. This includes a critical fix for relative path handling that previously caused silent failures.
  • Dead Code Detection (--dead-code): Introduced a heuristic-based feature to identify likely-dead functions (functions with no external callers).
  • Git Churn Ranking (--git-churn): Clones can now be annotated with Git commit churn scores and re-sorted by churn, highlighting frequently changed duplicate code.
  • Test↔Prod Leak Detection: Clones spanning both test and production files are now identified and tagged.
  • Safety Caps: Added --max-files and --max-pairs flags to prevent runaway scans on large codebases by failing fast if limits are exceeded.
  • Output Refinements: Implemented a --quiet flag to suppress info-level logs while preserving the scan report on stdout. The reporter also received extensive de-noising, including truncation of long lists, markdown badges for new clone attributes, and improved formatting for PR comments.
  • Comprehensive Testing & Docs: Added 55 new tests covering all new features, including security and performance aspects. The documentation (README, EXAMPLES.md) has been significantly updated with detailed recipes and a new demo GIF.

Concerns

  • The dead code detection is explicitly noted as a heuristic. While valuable, it's important to acknowledge that it might produce false positives due to dynamic language features (e.g., reflection, dynamic dispatch) or cross-package calls not easily detectable by token-based analysis. The current implementation includes a disclaimer in the report, which is good.
  • The Python function body detection heuristic in markPythonFunctions relies on line gaps to approximate indentation levels. While a reasonable approach without full AST parsing, it might not be perfectly robust for all Python code styles or complex nested structures.

Verdict

Approve


Code review performed by GEMINI - gemini-2.5-flash.

…s it

The dupehound bot comment on PRs was running the scan without --dead-code,
so the report had hotspots and clones but no dead-function section. Add
--dead-code to both scan invocations:

- The --verbose run that logs to CI output (useful for diagnostics)
- The --format md run that produces the PR comment (the user-visible one)

Also add --quiet to the markdown run to make intent explicit (the captured
artifact is the report on stdout, not the progress logs on stderr — the
flag protects against any future log line accidentally going to stdout).

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This Pull Request introduces a significant set of enhancements to dupehound, focusing on improving its utility, performance, and safety. Key features include:

  • Expanded File Filtering: Added support for ** glob patterns in --exclude and a new --include allowlist flag, providing more granular control over which files are scanned.
  • Diff-Aware Scanning (--since): A major rewrite of the --since flag transforms it from a post-detection filter into a true performance optimization. The detector now skips processing clone candidates entirely if they are not within files changed since the specified Git ref, leading to substantial speedups on large repositories.
  • Dead Code Detection (--dead-code): Implemented a heuristic-based detection for functions that appear to have no external callers, helping identify potentially unused code.
  • Git Churn Ranking (--git-churn): Clones can now be annotated with Git commit churn scores and re-sorted by churn, highlighting frequently changed duplicated code.
  • Suppression Mechanisms: Introduced a .dupehound-ignore file and inline //dupehound:ignore markers to suppress known or intentional duplicates across all 20 supported languages.
  • Safety Caps: Added --max-files and --max-pairs flags to prevent runaway scans on extremely large codebases by failing fast if resource limits are exceeded.
  • Reporter Improvements: Enhanced output readability with --top N truncation, separate listing for test↔prod clones, and visual badges (test↔prod, churn, suppressed) in Markdown reports. Also, progress logs are now throttled, and --quiet suppresses info logs while preserving the full report.
  • Binary File Skipping: Automatically skips binary files to prevent errors and improve scan efficiency.
  • Test Coverage: Significantly increased test coverage with 55 new tests, including comprehensive integration tests for all new features, edge cases, and security aspects.

Concerns

  • Heuristic Limitations: The dead code detection is explicitly noted as a heuristic, meaning it may produce false positives due to dynamic calls, reflection, or cross-package boundaries. This is an inherent limitation of static analysis without full program understanding, but it's well-documented.
  • Suppression False Positives: The matchesFilePath logic in services/suppression.go for file:line range suppression might be overly broad, potentially suppressing more than intended if a rule file path is a substring of another file's path (e.g., foo matching foobar.go).

Verdict

Request Changes

Confidence

90


Code review performed by GEMINI - gemini-2.5-flash.

The dupehound bot comment was a full scan on every PR. On a tiny repo
that's fine, but it has two real downsides:

1. The report shows clones in code the PR didn't touch, so reviewers
   have to mentally filter to find what's actually relevant to merge.
2. It doesn't dogfood --since — the perf rewrite shipped in this PR
   is exactly the pattern other projects should adopt for their own
   self-scans.

Switch both scan invocations to --since "origin/${BASE_REF}", using
github.base_ref so PRs targeting any branch (not just main) work
correctly. The base ref is hoisted into an env var first so the
workflow expression never lands inside a shell command directly.

Set fetch-depth: 0 on actions/checkout — --since calls
git rev-parse --verify which needs actual history, and the default
depth-1 checkout would silently trip the "could not get git diff,
falling back to full scan" warning path.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This extensive pull request introduces nine significant enhancements to dupehound, focusing on improved usability, performance, and reporting capabilities. Key features include:

  • Enhanced File Filtering: Added support for ** glob patterns in --exclude and introduced a new --include flag for allowlisting files, along with a mechanism to skip binary files.
  • Code Suppression: Implemented .dupehound-ignore files and inline //dupehound:ignore markers, allowing users to suppress known or intentional duplicates across all 20 supported languages.
  • Diff-Aware Scanning (--since): Rewrote the --since <ref> functionality to be a true performance optimization, where the detector itself skips work for clones entirely outside the changed files, rather than just post-filtering results. This significantly reduces scan times on large repositories with small diffs.
  • New Detection Capabilities: Introduced --dead-code for heuristic detection of unused functions, test↔prod span detection to identify duplication between test and production code, and --git-churn for ranking clones by commit activity.
  • Safety and Control: Added --max-files and --max-pairs as fail-fast safety caps to prevent runaway scans on large codebases, and a --quiet flag to suppress verbose logging while preserving the scan report.
  • Improved Reporting: The text and Markdown reporters received significant de-noising, including capping instance lists, adding inline badges (test↔prod, churn, suppressed), and wrapping dead function overflows in <details> blocks for cleaner PR comments.
  • Documentation and Testing: The PR includes a comprehensive EXAMPLES.md cookbook, an updated README.md, and 55 new unit and integration tests covering all new features, edge cases, and security aspects.

Concerns

  • Heuristic Limitations: The dead code detection and inline ignore marking for some languages (e.g., C-like languages in services/deadcode.go and services/tokenizer.go) rely on heuristics. While acknowledged in the documentation and code comments, these might lead to occasional false positives/negatives or brittle behavior in complex code structures. This is a known trade-off for a static analysis tool without full AST parsing.
  • --max-files Error Message Clarity: The error message for --max-files could be slightly more precise, as the include/exclude filters are applied before the cap check, meaning the suggested actions are already implicitly handled.
  • fileInScope Safety: The fileInScope helper in services/detector.go treats out-of-range indices as in-scope. While safe against panics, it could potentially mask an underlying indexing bug if the inScopeFiles slice is unintentionally shorter than the files slice.

Verdict

approve


Code review performed by GEMINI - gemini-2.5-flash.

The dead-code heuristic was reporting false positives for production
functions called only from excluded files. Specifically: dupehound's own
PR #24 self-scan (which excludes *_test.go via .dupehound.yml) flagged
BuildTokenizedFile and Detect as dead, even though both are called by
~30 test sites that the scanner had filtered out before tokenizing.

Root cause: findDeadFunctions was given a single file slice and used it
for both "where to look for definitions" AND "where to look for usages".
When the user excluded test files, those tokens never reached the
identifier-usage map, so any test-only helper became a false positive.

Fix: split the two responsibilities.

- findDeadFunctions(defFiles, usageFiles []TokenizedFile) — defFiles is
  the in-scope set the user wants findings reported for; usageFiles is
  the broader background set used only for the identifier-usage map.
- scanner.go builds the background set on demand: when --exclude or
  --include is active and --dead-code is on, it re-walks the scan path
  with NO filters via collectFiles(path, nil, nil, lang), tokenizes any
  files not already in the in-scope set, and appends them to usageFiles.
- The clone reports are unaffected — only the dead-code analysis gets
  the broader set. Excluded files still don't appear in clones / hotspots.

Verified on the live dupehound repo: pre-fix self-scan reported 2 dead
functions (BuildTokenizedFile, Detect — both false positives). Post-fix:
0 dead functions, which is the correct answer.

Tests added (+2):
- TestDeadCode_TestOnlyHelperNotFlagged: regression for the exact
  scenario that triggered on PR #24 — production function called only
  from a *_test.go file with `Exclude: ["*_test.go"]`. Pre-fix this
  reported the function as dead; post-fix it does not.
- TestDeadCode_GenuinelyDeadStillFlaggedWithExcludes: the matching
  positive barrier — the broader usage scan must NOT cause genuinely
  unreachable functions to be missed. A function with zero callers
  anywhere (including tests) is still flagged when excludes are active.

The new background tokenization is gated behind `len(opts.Exclude) > 0
|| len(opts.Include) > 0` so users with no filters pay zero extra cost.
For users with filters, the extra work is bounded by the file walk and
shares the same security envelope as the main collection path
(filepath.Walk + binary skip + same ignore rules).
@github-actions

github-actions Bot commented Apr 8, 2026

Copy link
Copy Markdown

dupehound scan results

Scanning diff since: origin/main (0 files changed)

Metric Value
Files scanned 17 / 17
Total lines 3588
Clones found 0
Duplicate lines 0 (0.0%)
New clones since origin/main 0

No duplicates found.


View full run · Scanned by dupehound

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This Pull Request introduces nine significant enhancements to dupehound, focusing on improving its utility, performance, and user experience. Key features include:

  • Diff-aware scanning (--since): A major rewrite of the --since flag to perform true cost reduction by skipping detection work on unchanged files, rather than just post-filtering results. This significantly improves performance on large repositories when scanning only recent changes.
  • Dead code detection (--dead-code): A new heuristic to identify functions that appear to have no external callers, helping developers find potentially unused code.
  • Inline suppression (.dupehound-ignore and //dupehound:ignore): Allows users to suppress known duplicate code blocks or files using a .dupehound-ignore file or inline comments, reducing noise in reports.
  • Git churn ranking (--git-churn): Annotates clones with git commit churn scores and re-sorts them, highlighting frequently changed duplicate code.
  • Test↔Prod span detection: Identifies clones that span both test and production code, which can be a signal for problematic duplication.
  • Safety caps (--max-files, --max-pairs): Introduces fail-fast mechanisms to prevent runaway scans on excessively large codebases or during fuzzy detection, improving stability for unattended CI/CD runs.
  • Output de-noising (--quiet, --top): Enhancements to the reporter to provide more concise and actionable output, including capping instance lists, aggregating fuzzy detection logs, and adding visual badges to Markdown reports.
  • Improved file filtering (--include, ** glob support): Adds an --include allowlist and extends glob pattern matching to support ** for more flexible file selection.
  • Binary file skipping: Automatically skips binary files to prevent errors and improve scan efficiency.

The PR also includes extensive new tests (55 additions), updated documentation, and a new demo GIF, ensuring robustness and clarity for the new features.

Concerns

  • Heuristic limitations in dead code detection: The PR explicitly states that dead function detection is a heuristic and may produce false positives (e.g., due to reflection, cross-package calls). While this is acknowledged, users should be aware of this limitation and interpret results with caution. The current implementation of markIgnoredBlocks in services/tokenizer.go also relies on a markerLine+3 heuristic which might be brittle if there are many blank lines or comments between the ignore marker and the function definition, potentially leading to false negatives for suppression.

Verdict

Approve


Code review performed by GEMINI - gemini-2.5-flash.

Comment thread services/tokenizer.go
continue
}
// Find the first token that is in-function on a subsequent line.
markerLine := tokens[i].Line

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] The markerLine+3 heuristic here might be a bit brittle. If there are more than 3 lines of non-token content (e.g., blank lines, comments) between the //dupehound:ignore marker and the actual function body, it might prematurely break the loop and miss marking the function as ignored. While inFunc[j] is the primary guard, this could lead to false negatives for suppression in some edge cases. Consider if a more robust way to find the start of the function body after a marker is needed, or if this is an acceptable limitation for a heuristic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: support ** recursive glob patterns in --exclude

1 participant

⌂ 换网址