Repository navigation
feat(#15-#23): nine enhancements + diff-aware --since rewrite + safety caps - #24
Conversation
Add new fields to Report, Clone, CloneInstance structs for binary skip, suppression, test-prod span, dead code, diff-aware scanning, and churn. Introduce --include, --top, --since, --show-suppressed, --dead-code, --git-churn, --churn-days flags. Extend ScanConfig with include/top. Add gobwas/glob for ** glob support. Add coverage.out to .gitignore and .structlint.yaml ignore list.
Binary file skip (#16): before tokenizing, read first 512 bytes and skip files containing a null byte; increment skipped_files in report. Double-star glob in --exclude (#15): replace filepath.Match with gobwas/glob for ** support. Patterns without ** continue to match basenames as before. Include filter (#17): add --include flag using same ** glob support. If set, only files matching at least one include pattern are scanned. Without --include all files are scanned (backward compatible). Add BuildTokenizedFileWithIgnore for inline dupehound:ignore annotation support. Extend TokenizedFile with Ignored field. Add markIgnoredBlocks to tokenizer.
…ode, test-prod, churn Clone suppression (#18): .dupehound-ignore file support with hash prefix, path glob (** supported), and file:line range rules. Inline //dupehound:ignore annotation excludes function bodies from detection. --show-suppressed flag. Top-N output (#19): --top flag limits text/md output (0=all). JSON/SARIF always return full output. Diff-aware scanning (#20): --since <ref> filters clones to those touching lines changed since the git ref. Adds new_clones to report JSON. Dead code detection (#21): --dead-code flag identifies functions with no external callers using token-based heuristic analysis. Test↔prod clone annotation (#22): annotate clones spanning test and non-test files with TestProdSpan=true and IsTest=true on instances. Shown as separate section in text/md output. SARIF uses ruleId DUPE002 for these. Git churn (#23): --git-churn re-sorts clones by commit activity (--churn-days window, default 90 days). Annotates instances with FileCommits.
Scanner: invoke suppression, test-prod annotation, diff filtering, churn scoring, and dead code detection in the scan pipeline. Binary files increment skipped_files counter. Reporter: add test-prod section in text/md output, --top-N truncation, dead function report with heuristic disclaimer, since-diff header, churn score/commit labels. SARIF uses DUPE002 for test-prod clones. Scan action: wire all new CLI flags to ScanOptions and FormatOptions. Exit code 1 when --since finds new clones or --dead-code finds dead functions (--exit-zero still overrides). --min-duplication applies to new-clone percentage when --since is active.
Binary skip: isBinaryData unit tests, scan skips binary files and counts them in skipped_files. Glob/include: matchGlob with **, matchesIncludes, scan with --exclude vendor/**, --exclude **/*_test.go, --include **/*.go, combined include+exclude. Top-N: topNWithOpt unit test, text output truncated at --top 3, --top -1 shows all, JSON unaffected. Test-prod: isTestFile for all languages, annotateTestProdSpan mixed/ both-prod/both-test, full scan annotation. Suppression: by hash, path glob, file:line range, --show-suppressed, no ignore file, loading from actual file, parseIgnoreLine parsing. Dead code: called function not dead, unused function detected, flag absent means no output, JSON/text output format. Diff filtering: parseDiffHunks, filterClonesInDiff, overlapsChangedLines, invalid ref graceful failure. Churn: sortByChurn ordering, tie-break, git-churn absent means default sort, JSON/text output with churn fields. Inline ignore: //dupehound:ignore marks Go tokens, # dupehound:ignore marks Python tokens, isIgnoreComment variants.
- Reject --since refs starting with '-' or containing shell-unsafe characters (space, semicolon, backtick, dollar) to prevent git from interpreting user input as option flags (e.g. --output=, --upload-pack=) - Add TestValidateGitRef covering valid refs (origin/main, HEAD~1, tags) and invalid inputs (flags, empty string, shell metacharacters) - Fix matchGlob: remove redundant second glob.Compile that tested filepath.Base against a ** pattern — semantically wrong since vendor/** can never match "user.go", and compiled the same pattern twice
Replace character-allowlist validation with resolveGitRef() which calls `git rev-parse --verify <ref>` and returns the full 40-char hex SHA. All downstream git diff calls now use only the SHA — a hex string that can never be misinterpreted as a flag regardless of what the user passes. The '-' prefix guard is kept as a fast-path to reject obvious flag-shaped inputs before forking a git process at all. Add tests: - TestResolveGitRef_DashPrefix: flag-shaped inputs rejected without git - TestResolveGitRef_ValidSHA: HEAD resolves to a safe hex-only string
Tests added (+34, total 256 passing): - #17 strict negative test: --include must exclude non-matching files from TotalFiles, ScannedFiles, FileStats — not just from clones - #23 real-git-repo end-to-end churn test: builds a temp repo with known commit history (a.go x4, b.go x1) and asserts FileCommits + ChurnScore are correct, plus the churn-descending sort invariant - #18 parametrized inline-ignore across all 20 supported languages and 4 comment styles (//, #, --, /* */); negative cases for over-eager match; Python end-to-end exclusion test (different function-body marker than Go); strengthened TestMarkIgnoredBlocks_ExcludesFromDetection from a no-op TotalFiles assertion to a real cross-file exclusion check Reporter (text + markdown) noise fixes: - Remove unconditional "All clones:" dump that defeated --top in text format - Stop double-listing test↔prod clones (top section now iterates normalClones) - Cap test↔prod section by --top with overflow note - Cap clone instance lists to 6 (was unbounded — clone with 28 instances dumped 28 lines); --verbose still shows all - Add test↔prod / churn / suppressed badges to markdown clone summaries (PR comments previously lost churn entirely) - Per-instance commit counts in markdown locations - Cap dead-functions list in both formats; markdown overflow in details - Fix markdown early-return on TotalClones==0 dropping dead functions Detector log throttling (CI / hook UX): - Aggregate fuzzy bucket truncated warnings into a single summary line - Throttle fuzzy detection progress to ~20 emissions max regardless of pair count (was every 10k → thousands of lines/sec on fast machines) Trial on cli/cli (816 Go files): output went from thousands of stderr log lines to 24, and clone instance lists are now bounded.
EXAMPLES.md is a copy-pasteable cookbook organized in collapsible sections for the features added in #15-#23. Topics: pre-commit hooks (lefthook + plain git), GH Actions PR comments, diff-aware --since, include/exclude globs, .dupehound-ignore + inline suppression, test↔prod leak detection, git churn ranking, dead-function detection, safe-mode profiles for large monorepos, output formats, and a real-world trial on cli/cli with measured times and findings. README.md adds a Contents TOC, refreshes the flag list with all the new options (--include, --since, --top, --git-churn, --churn-days, --dead-code, --show-suppressed), and adds a "More examples & use cases" section that deep-links into EXAMPLES.md so the README itself stays terse.
The --since flag was a post-detection filter: it ran a full clone scan, then dropped clones whose instances didn't touch the diff. On a 800-file repo with a 30-file PR diff this paid the full O(N²) fuzzy detection cost for ~3% of the files we cared about. This commit threads an InScopeFiles []bool through DetectOptions and uses it in two places: - detectExact: when emitting a clone group, skip it (and mark its positions covered) if none of the participating files are in scope. - detectFuzzy: when building candidate pairs, skip pairs where neither block is in a file marked in scope. This is the dominant cost win — it cuts both the pair count and the dedup map size. The scanner resolves the in-scope file set BEFORE detection by calling getChangedLines once and reusing the result for the existing line-precise NewClones post-filter (one fewer git invocation as a side benefit). Path normalization fix: getChangedLines returns absolute paths, but tokenized files carry whatever form the walker produced. With --path . the lookup never matched and --since silently produced 0 clones. The scanner now rekeys the changedLines map by the same form as tokenizedFiles[i].Path, so --since works correctly with both relative and absolute --path. Measured on cli/cli (816 Go files): full scan 2:21 → --since HEAD~10 23s, same 211 in-scope clones, identical regardless of relative/absolute --path. Tests added (8 total): Unit (DetectWithOptions, no scanner): - TestDetector_InScopeFilter_OnlyReportsClonesTouchingScope: a 3-file scenario where only one file is in scope; clone is emitted because at least one instance touches the scope set - TestDetector_InScopeFilter_DropsAllOutOfScopeClones: empty scope ⇒ 0 clones (the fast path that --since relies on) - TestDetector_InScopeFilter_NilPreservesOriginalBehavior: regression guard against anyone making nil mean "nothing in scope" - TestDetector_InScopeFilter_FuzzyPath: scope filter is honored on the type-3 path, not just the exact path - TestDetector_InScopeFilter_OutOfRangeIndexIsSafe: defensive — short scope slice doesn't panic; out-of-range indices are treated as in scope End-to-end (real git repo via Scanner.Scan): - TestScanner_Since_RealRepo_FewerCloneEvaluations: builds a temp repo with known commit history, asserts --since produces ≤ as many clones as full scan AND the in-diff file is surfaced in NewClones - TestScanner_Since_NoChangesNoClones: --since HEAD on a repo with no diff produces zero clones regardless of how much duplication exists (the safety guarantee that makes --since usable in pre-commit hooks) - TestScanner_Since_RelativePath: regression for the silent path-form bug — runs --since with --path "." and asserts the diff is picked up
…d use Three safety nets for running dupehound on huge monorepos and inside hooks without OOM or stderr noise. --max-files N (default 0 = no cap) Hard cap checked after include/exclude filtering. If the file walk collected more files than the cap, the scanner returns ErrTooManyFiles with an actionable message naming both the count and the cap. Catches misconfigured --path before any tokenization runs. --max-pairs N (default 0 = no cap) Hard cap on fuzzy candidate pairs. The detector aborts type-3 detection (returning whatever exact clones it had) before the dedup map can grow past the cap. Surfaces a WARN-level log explaining the fix: lower --max-bucket, raise --max-pairs, or disable fuzzy with --similarity 1.0. Prevents the multi-GB pair dedup map on pathological repos. --quiet, -q Maps to zerolog warn level. INFO-level progress logs (starting scan, scope summary, fuzzy progress) are suppressed; warnings, errors, and the scan report itself are unaffected. The report goes to stdout via fmt.Print, so a failing pre-commit hook still prints the full clone listing — file paths, line ranges, and code preview — exactly what a developer or LLM-based reviewer needs to fix the problem. --verbose wins over --quiet (more output is the safer default for ambiguous intent). Tests added (8 total): services/limits_test.go (6): - TestScanner_MaxFiles_FailsFast: feature test — error wraps ErrTooManyFiles, message mentions actual numbers and remediation flags - TestScanner_MaxFiles_Disabled: regression — MaxFiles=0 ⇒ no cap - TestScanner_MaxFiles_AtBoundary: off-by-one barrier (= passes, +1 fails) - TestScanner_MaxFiles_AppliedAfterIncludeFilter: cap is checked against the post-filter file count, not the raw walk - TestDetector_MaxPairs_AbortsFuzzy: tight cap on a many-near-miss corpus produces a strict subset of the baseline and contains zero type-3 clones - TestDetector_MaxPairs_Zero: regression — MaxPairs=0 ⇒ no cap integration/cli_test.go (2): - TestQuiet_SuppressesInfoLogsButKeepsReport: end-to-end — baseline scan emits "starting scan" on stderr, --quiet does not, but stdout still contains "Clones found" and "Top clones" (the failure-reason guarantee) - TestQuiet_VerboseWinsOverQuiet: precedence regression — both flags ⇒ verbose level Security review of all changes in this commit and the preceding --since commit: no new attack surface. The new flags are pure integer comparisons and a log level mapping. The --since path normalization uses filepath.Clean and filepath.Abs (deterministic stdlib functions) on paths the scanner already trusted. The git ref still flows through resolveGitRef which rejects dash-prefixed refs and converts to a hex SHA before any subsequent git invocation. go vet clean.
EXAMPLES.md: - Diff-aware section now reflects that --since is a true cost reducer, not a post-filter; includes the cli/cli measurement (2:21 → 23s) - Safe-mode "big picture" rewritten with the full safety knob hierarchy: --similarity 1.0, --max-pairs, --max-files, --max-bucket, --since, --include — in order of impact - Profile A (pre-commit) and Profile B (CI on PR diff) updated to add --quiet and to use --since for the CI profile - New Profile D for paranoid unattended scans with all hard caps applied - New "Hook output and AI readability" section explaining the stream contract: --quiet only filters info logs on stderr, the clone report on stdout (the failure reason) is unaffected, so failing hooks remain fully usable for developers and LLM-based reviewers - Quick reference table and TOC link to all new sections README.md: - Flag list refreshed with --max-files, --max-pairs, --quiet - --since description updated to mention it's a real cost reduction - Deep-link list points to the new safe-mode and hook-output sections
Single hero demo at the top of EXAMPLES.md and README.md, recorded against a real public sample app (axeforging/tacomex-8bit-shop — React + Fastify, 69 source files, ~14k LOC). Two acts: 1. Plain scan finds 163 clones / 20% duplication. Top hit is a 38-line type-2 block repeated 11 times in backend/src/seeds/index.ts (the taco data) — a clear extract-helper opportunity, not a fake fixture. 2. --dead-code surfaces real candidates: closeConnection, optionalAuth, requireAdmin — actual functions that look genuinely dead in the repo. The demo is reproducible. docs/demo/ contains: - cast.yaml: declarative spec for the recording (tracked) - setup.sh: idempotent — clones tacomex into /tmp/dupehound-demo-repo (tracked) - README.md: regeneration instructions (tracked) - dupehound-demo.gif: 191 KB final render (tracked) Intermediates that should not be committed are in .gitignore: - docs/demo/*.cast (asciinema raw recording) - docs/demo/.bin/ (any locally-built binary) structlint: allow docs/** paths and *.gif / *.sh extensions; ignore the .cast intermediate so it doesn't trip the structure validator after a local re-record. Why one hero instead of one-per-recipe: per-recipe GIFs go stale on every flag/output change (we just shipped --quiet, --max-files, --max-pairs, and the --since rewrite — every recipe recording from yesterday would already need re-recording). The text recipes in EXAMPLES.md remain the canonical copy-paste source; the hero GIF is the "what does this thing do" entry point that benefits from being seen, not read.
Regenerating the GIF was a two-step dance: bash docs/demo/setup.sh, then run cast. Move the (idempotent) clone into a `bash:` scene at the top of cast.yaml so the whole thing is one command: python3 path/to/cast.py docs/demo/cast.yaml The clone uses `test -d /tmp/dupehound-demo-repo || git clone …` so re-runs short-circuit instantly. setup.sh deleted; structlint config no longer needs the *.sh allowance. docs/demo/README.md updated with the new single-command workflow. Re-recorded from a clean state to verify the inlined clone path works. GIF size unchanged (~188 KB), both acts (dedup + dead-code) intact.
StructLint — All checks passed72 rules validated against
|
There was a problem hiding this comment.
Core Changes
This Pull Request introduces a suite of significant enhancements to dupehound, focusing on improved usability, performance, and new detection capabilities. Key features include:
- Advanced File Filtering: Implemented
**glob support for--excludeand a new--includeallowlist, providing more granular control over scanned files. - Code Suppression: Added support for
.dupehound-ignorefiles and inline//dupehound:ignorecomments across all 20 supported languages, allowing users to suppress known or intentional duplicates. - Diff-Aware Scanning (
--since): Rewrote the--sincefunctionality to be a true performance optimization, where the detector actively skips work for clones not touching changed files. This includes a critical fix for relative path handling that previously caused silent failures. - Dead Code Detection (
--dead-code): Introduced a heuristic-based feature to identify likely-dead functions (functions with no external callers). - Git Churn Ranking (
--git-churn): Clones can now be annotated with Git commit churn scores and re-sorted by churn, highlighting frequently changed duplicate code. - Test↔Prod Leak Detection: Clones spanning both test and production files are now identified and tagged.
- Safety Caps: Added
--max-filesand--max-pairsflags to prevent runaway scans on large codebases by failing fast if limits are exceeded. - Output Refinements: Implemented a
--quietflag to suppress info-level logs while preserving the scan report on stdout. The reporter also received extensive de-noising, including truncation of long lists, markdown badges for new clone attributes, and improved formatting for PR comments. - Comprehensive Testing & Docs: Added 55 new tests covering all new features, including security and performance aspects. The documentation (README, EXAMPLES.md) has been significantly updated with detailed recipes and a new demo GIF.
Concerns
- The dead code detection is explicitly noted as a heuristic. While valuable, it's important to acknowledge that it might produce false positives due to dynamic language features (e.g., reflection, dynamic dispatch) or cross-package calls not easily detectable by token-based analysis. The current implementation includes a disclaimer in the report, which is good.
- The Python function body detection heuristic in
markPythonFunctionsrelies on line gaps to approximate indentation levels. While a reasonable approach without full AST parsing, it might not be perfectly robust for all Python code styles or complex nested structures.
Verdict
Approve
Code review performed by GEMINI - gemini-2.5-flash.
…s it The dupehound bot comment on PRs was running the scan without --dead-code, so the report had hotspots and clones but no dead-function section. Add --dead-code to both scan invocations: - The --verbose run that logs to CI output (useful for diagnostics) - The --format md run that produces the PR comment (the user-visible one) Also add --quiet to the markdown run to make intent explicit (the captured artifact is the report on stdout, not the progress logs on stderr — the flag protects against any future log line accidentally going to stdout).
There was a problem hiding this comment.
Core Changes
This Pull Request introduces a significant set of enhancements to dupehound, focusing on improving its utility, performance, and safety. Key features include:
- Expanded File Filtering: Added support for
**glob patterns in--excludeand a new--includeallowlist flag, providing more granular control over which files are scanned. - Diff-Aware Scanning (
--since): A major rewrite of the--sinceflag transforms it from a post-detection filter into a true performance optimization. The detector now skips processing clone candidates entirely if they are not within files changed since the specified Git ref, leading to substantial speedups on large repositories. - Dead Code Detection (
--dead-code): Implemented a heuristic-based detection for functions that appear to have no external callers, helping identify potentially unused code. - Git Churn Ranking (
--git-churn): Clones can now be annotated with Git commit churn scores and re-sorted by churn, highlighting frequently changed duplicated code. - Suppression Mechanisms: Introduced a
.dupehound-ignorefile and inline//dupehound:ignoremarkers to suppress known or intentional duplicates across all 20 supported languages. - Safety Caps: Added
--max-filesand--max-pairsflags to prevent runaway scans on extremely large codebases by failing fast if resource limits are exceeded. - Reporter Improvements: Enhanced output readability with
--top Ntruncation, separate listing for test↔prod clones, and visual badges (test↔prod, churn, suppressed) in Markdown reports. Also, progress logs are now throttled, and--quietsuppresses info logs while preserving the full report. - Binary File Skipping: Automatically skips binary files to prevent errors and improve scan efficiency.
- Test Coverage: Significantly increased test coverage with 55 new tests, including comprehensive integration tests for all new features, edge cases, and security aspects.
Concerns
- Heuristic Limitations: The dead code detection is explicitly noted as a heuristic, meaning it may produce false positives due to dynamic calls, reflection, or cross-package boundaries. This is an inherent limitation of static analysis without full program understanding, but it's well-documented.
- Suppression False Positives: The
matchesFilePathlogic inservices/suppression.gofor file:line range suppression might be overly broad, potentially suppressing more than intended if a rule file path is a substring of another file's path (e.g.,foomatchingfoobar.go).
Verdict
Request Changes
Confidence
90
Code review performed by GEMINI - gemini-2.5-flash.
The dupehound bot comment was a full scan on every PR. On a tiny repo
that's fine, but it has two real downsides:
1. The report shows clones in code the PR didn't touch, so reviewers
have to mentally filter to find what's actually relevant to merge.
2. It doesn't dogfood --since — the perf rewrite shipped in this PR
is exactly the pattern other projects should adopt for their own
self-scans.
Switch both scan invocations to --since "origin/${BASE_REF}", using
github.base_ref so PRs targeting any branch (not just main) work
correctly. The base ref is hoisted into an env var first so the
workflow expression never lands inside a shell command directly.
Set fetch-depth: 0 on actions/checkout — --since calls
git rev-parse --verify which needs actual history, and the default
depth-1 checkout would silently trip the "could not get git diff,
falling back to full scan" warning path.
There was a problem hiding this comment.
Core Changes
This extensive pull request introduces nine significant enhancements to dupehound, focusing on improved usability, performance, and reporting capabilities. Key features include:
- Enhanced File Filtering: Added support for
**glob patterns in--excludeand introduced a new--includeflag for allowlisting files, along with a mechanism to skip binary files. - Code Suppression: Implemented
.dupehound-ignorefiles and inline//dupehound:ignoremarkers, allowing users to suppress known or intentional duplicates across all 20 supported languages. - Diff-Aware Scanning (
--since): Rewrote the--since <ref>functionality to be a true performance optimization, where the detector itself skips work for clones entirely outside the changed files, rather than just post-filtering results. This significantly reduces scan times on large repositories with small diffs. - New Detection Capabilities: Introduced
--dead-codefor heuristic detection of unused functions,test↔prodspan detection to identify duplication between test and production code, and--git-churnfor ranking clones by commit activity. - Safety and Control: Added
--max-filesand--max-pairsas fail-fast safety caps to prevent runaway scans on large codebases, and a--quietflag to suppress verbose logging while preserving the scan report. - Improved Reporting: The text and Markdown reporters received significant de-noising, including capping instance lists, adding inline badges (test↔prod, churn, suppressed), and wrapping dead function overflows in
<details>blocks for cleaner PR comments. - Documentation and Testing: The PR includes a comprehensive
EXAMPLES.mdcookbook, an updatedREADME.md, and 55 new unit and integration tests covering all new features, edge cases, and security aspects.
Concerns
- Heuristic Limitations: The dead code detection and inline ignore marking for some languages (e.g., C-like languages in
services/deadcode.goandservices/tokenizer.go) rely on heuristics. While acknowledged in the documentation and code comments, these might lead to occasional false positives/negatives or brittle behavior in complex code structures. This is a known trade-off for a static analysis tool without full AST parsing. --max-filesError Message Clarity: The error message for--max-filescould be slightly more precise, as the include/exclude filters are applied before the cap check, meaning the suggested actions are already implicitly handled.fileInScopeSafety: ThefileInScopehelper inservices/detector.gotreats out-of-range indices as in-scope. While safe against panics, it could potentially mask an underlying indexing bug if theinScopeFilesslice is unintentionally shorter than thefilesslice.
Verdict
approve
Code review performed by GEMINI - gemini-2.5-flash.
The dead-code heuristic was reporting false positives for production functions called only from excluded files. Specifically: dupehound's own PR #24 self-scan (which excludes *_test.go via .dupehound.yml) flagged BuildTokenizedFile and Detect as dead, even though both are called by ~30 test sites that the scanner had filtered out before tokenizing. Root cause: findDeadFunctions was given a single file slice and used it for both "where to look for definitions" AND "where to look for usages". When the user excluded test files, those tokens never reached the identifier-usage map, so any test-only helper became a false positive. Fix: split the two responsibilities. - findDeadFunctions(defFiles, usageFiles []TokenizedFile) — defFiles is the in-scope set the user wants findings reported for; usageFiles is the broader background set used only for the identifier-usage map. - scanner.go builds the background set on demand: when --exclude or --include is active and --dead-code is on, it re-walks the scan path with NO filters via collectFiles(path, nil, nil, lang), tokenizes any files not already in the in-scope set, and appends them to usageFiles. - The clone reports are unaffected — only the dead-code analysis gets the broader set. Excluded files still don't appear in clones / hotspots. Verified on the live dupehound repo: pre-fix self-scan reported 2 dead functions (BuildTokenizedFile, Detect — both false positives). Post-fix: 0 dead functions, which is the correct answer. Tests added (+2): - TestDeadCode_TestOnlyHelperNotFlagged: regression for the exact scenario that triggered on PR #24 — production function called only from a *_test.go file with `Exclude: ["*_test.go"]`. Pre-fix this reported the function as dead; post-fix it does not. - TestDeadCode_GenuinelyDeadStillFlaggedWithExcludes: the matching positive barrier — the broader usage scan must NOT cause genuinely unreachable functions to be missed. A function with zero callers anywhere (including tests) is still flagged when excludes are active. The new background tokenization is gated behind `len(opts.Exclude) > 0 || len(opts.Include) > 0` so users with no filters pay zero extra cost. For users with filters, the extra work is bounded by the file walk and shares the same security envelope as the main collection path (filepath.Walk + binary skip + same ignore rules).
dupehound scan results
No duplicates found. View full run · Scanned by dupehound |
There was a problem hiding this comment.
Core Changes
This Pull Request introduces nine significant enhancements to dupehound, focusing on improving its utility, performance, and user experience. Key features include:
- Diff-aware scanning (
--since): A major rewrite of the--sinceflag to perform true cost reduction by skipping detection work on unchanged files, rather than just post-filtering results. This significantly improves performance on large repositories when scanning only recent changes. - Dead code detection (
--dead-code): A new heuristic to identify functions that appear to have no external callers, helping developers find potentially unused code. - Inline suppression (
.dupehound-ignoreand//dupehound:ignore): Allows users to suppress known duplicate code blocks or files using a.dupehound-ignorefile or inline comments, reducing noise in reports. - Git churn ranking (
--git-churn): Annotates clones with git commit churn scores and re-sorts them, highlighting frequently changed duplicate code. - Test↔Prod span detection: Identifies clones that span both test and production code, which can be a signal for problematic duplication.
- Safety caps (
--max-files,--max-pairs): Introduces fail-fast mechanisms to prevent runaway scans on excessively large codebases or during fuzzy detection, improving stability for unattended CI/CD runs. - Output de-noising (
--quiet,--top): Enhancements to the reporter to provide more concise and actionable output, including capping instance lists, aggregating fuzzy detection logs, and adding visual badges to Markdown reports. - Improved file filtering (
--include,**glob support): Adds an--includeallowlist and extends glob pattern matching to support**for more flexible file selection. - Binary file skipping: Automatically skips binary files to prevent errors and improve scan efficiency.
The PR also includes extensive new tests (55 additions), updated documentation, and a new demo GIF, ensuring robustness and clarity for the new features.
Concerns
- Heuristic limitations in dead code detection: The PR explicitly states that dead function detection is a heuristic and may produce false positives (e.g., due to reflection, cross-package calls). While this is acknowledged, users should be aware of this limitation and interpret results with caution. The current implementation of
markIgnoredBlocksinservices/tokenizer.goalso relies on amarkerLine+3heuristic which might be brittle if there are many blank lines or comments between the ignore marker and the function definition, potentially leading to false negatives for suppression.
Verdict
Approve
Code review performed by GEMINI - gemini-2.5-flash.
| continue | ||
| } | ||
| // Find the first token that is in-function on a subsequent line. | ||
| markerLine := tokens[i].Line |
There was a problem hiding this comment.
[SUGGESTION] The markerLine+3 heuristic here might be a bit brittle. If there are more than 3 lines of non-token content (e.g., blank lines, comments) between the //dupehound:ignore marker and the actual function body, it might prematurely break the loop and miss marking the function as ignored. While inFunc[j] is the primary guard, this could lead to false negatives for suppression in some edge cases. Consider if a more robust way to find the start of the function body after a marker is needed, or if this is an acceptable limitation for a heuristic.
Closes #15, #16, #17, #18, #19, #20, #21, #22, #23.
Summary
This branch implements all nine enhancement issues, then layers on a perf rewrite of
--since, three safety knobs for unattended use (--max-files,--max-pairs,--quiet), reporter de-noising, and a hero demo GIF. 272 tests passing (started the branch at 217),go vetclean, lefthook (gofumpt + structlint + govet + golangci-lint + commitlint) green on every commit.Issues addressed
**glob in--exclude--includeallowlist.dupehound-ignore+ inlinedupehound:ignoremarkers (all 20 languages)--top Noutput verbosity--since <ref>diff-aware scanning (now a real cost reducer)--dead-codeheuristic--git-churnrankingWhat's in the branch beyond the bare features
Reporter de-noising (
fix(reporter))All clones:dump that was silently undoing--topnormalClones)... N more locations(was unbounded — clone with 28 instances dumped 28 lines on real repos)<details>--sinceperf rewrite (feat(detector))--sincewas a post-detection filter — paid the full O(N²) fuzzy cost on every file. ThreadedInScopeFiles []boolthroughDetectOptionsso the exact and fuzzy detectors skip work for clones whose instances are all in unchanged filesdupehound scan --path . --since mainused to always report 0 clones (relative-vs-absolute path mismatch in the lookup map)cli/cli(816 Go files): full scan 2:21 →--since HEAD~1023s, ~6× faster, identical correctness, works with both relative and absolute--pathSafety knobs for unattended use (
feat(safety))--max-files N— fails fast in the file collector withErrTooManyFilesand an actionable error message--max-pairs N— aborts fuzzy detection (returning exact-only + a WARN log) before allocating the multi-GB pair dedup map--quiet, -q— maps to zerolog warn level. INFO-level progress logs are silenced; the scan report on stdout is unaffected, so failing pre-commit hooks still print the full clone listing — exactly what a developer or LLM-based reviewer needs to act on the failureDocumentation
EXAMPLES.mdcookbook with 12 collapsible recipes, copy-pasteable, all anchors verifiedEXAMPLES.mddocs/demo/dupehound-demo.gif, 188 KB) recorded against realaxeforging/tacomex-8bit-shopshowing dedup + dead-code in action. Spec iscast.yaml(declarative, idempotent — clones tacomex on first run, short-circuits after); regenerable with one command. Intermediates gitignored, only the final GIF is tracked.Test additions (217 → 272, +55 meaningful tests)
Each new feature has regression, feature, and barrier (negative / edge / security) tests:
--include— non-matching files absent fromTotalFiles/ScannedFiles/FileStats, not just from clonesFileCommits, totalChurnScore, and the descending sort invariant--sincedetector scope: 8 tests covering exact path, fuzzy path, nil-preserves-behavior, out-of-range safety, real-repo end-to-end, no-changes-no-clones barrier, relative-path regression--quiet: 2 end-to-end integration tests proving stderr is silenced AND the scan report on stdout is preserved (the failure-reason guarantee for hooks)TestResolveGitRef_DashPrefixandTestResolveGitRef_ValidSHAfor the--sinceref injection defenseSecurity review
No new attack surface introduced. Specifically:
--sincepath normalization usesfilepath.Cleanandfilepath.Abs(deterministic stdlib) on paths the scanner already trustedresolveGitRefwhich rejects dash-prefixed refs and converts to a hex SHA before any subsequent git invocation. Two-layer defense (dash-prefix rejection + SHA-only usage downstream)getChangedLinesis now reused instead of called twicego vetclean, all pre-commit hooks greenKnown follow-ups (intentionally out of scope)
timeout 10m dupehound …wrapper documented in EXAMPLES.md--stagedis still a post-filter (only--sincewas rewritten). Pair with--similarity 1.0for cheap staged scans, or use--since HEAD --stagedtogether--quiettest. Sort tie-breaker isn't fully deterministic. Cosmetic but noticeable in run-to-run diffsTest plan
🤖 Generated with Claude Code