Repository navigation
feat: fuzzy bucket handling, actionable output, and markdown format - #11
Conversation
Replace the hard >100 skip with a cap at 500 blocks per mini-hash bucket. Blocks beyond the cap are truncated (not skipped entirely), and a debug log message is emitted so users know when it happens. This ensures valid type-3 clones in moderately large buckets are still detected.
StructLint — All checks passed48 rules validated against
|
There was a problem hiding this comment.
Core Changes
This pull request addresses a limitation in the detectFuzzy function where mini-hash buckets exceeding 100 entries were entirely skipped, potentially leading to missed type-3 (near-miss) clones in repetitive codebases. The change introduces a maxBucketSize constant (set to 500) and modifies the logic to truncate these large buckets to this maximum size instead of skipping them. This ensures that a significant portion of potential clone pairs within large buckets are still considered for Jaccard similarity evaluation. A debug log message is now emitted when a bucket is truncated, providing visibility into this behavior.
Concerns
While the maxBucketSize of 500 is a reasonable heuristic to balance performance and detection completeness, it is still an arbitrary cap. In extremely large buckets (e.g., thousands of entries), it's theoretically possible that the most relevant or highest-similarity clone pairs might exist beyond the first 500 entries and thus be missed. However, this is a necessary trade-off to prevent O(n^2) performance degradation, and the current approach is a significant improvement over skipping these buckets entirely.
Verdict
Approve
Code review performed by GEMINI - gemini-2.5-flash.
Add test verifying large buckets still produce type-3 clones. Upgrade truncation log from debug to warn so users see it without --verbose, since it affects result completeness. Improve log message with actionable context (total, evaluated, skipped counts).
There was a problem hiding this comment.
Core Changes
This Pull Request significantly improves the fuzzy (type-3) clone detection logic in services/detector.go. Previously, mini-hash buckets exceeding 100 blocks were entirely skipped, leading to missed near-miss clones in highly repetitive codebases. This change replaces the hard skip with a truncation mechanism, processing up to maxBucketSize (500) blocks per mini-hash bucket. A warning log is now emitted when a bucket is truncated, providing visibility into potential missed clones due to the performance cap.
Concerns
- Performance Trade-off: While the
maxBucketSizeof 500 is a reasonable cap to preventO(n^2)blowup, processing up to 125,000 pairs for a single mini-hash bucket could still be computationally intensive in extreme cases. The warning log helps identify when this truncation occurs, but users should be aware of this trade-off between completeness and performance for very large, highly repetitive codebases.
Verdict
Approve: The changes are a clear improvement over the previous behavior, addressing a significant limitation in fuzzy clone detection. The introduction of a configurable cap with logging provides a good balance between performance and accuracy. The new test case adequately covers the intended behavior.
Code review performed by GEMINI - gemini-2.5-flash.
…ogging Bump default fuzzy bucket cap from 500 to 5000 (~12.5M pairs, still fast). Add --max-bucket flag so users can tune or remove the limit. Add progress logging: info-level for >1000 pairs, periodic updates every 10k pairs.
There was a problem hiding this comment.
Core Changes
This Pull Request addresses a critical issue where type-3 (near-miss) clones were being silently missed in dupehound when mini-hash buckets grew too large. Previously, any bucket with more than 100 blocks was entirely skipped, leading to false negatives in clone detection for repetitive codebases.
The core changes include:
- Introduced
MaxBucketoption: A newMaxBucketinteger option has been added toScanOptionsandDetectOptionsto control the maximum number of candidate blocks processed per fuzzy hash bucket. This allows users to tune the trade-off between thoroughness and performance for type-3 detection. - Replaced hard skip with truncation: The
detectFuzzyfunction no longer skips large buckets entirely. Instead, it truncates buckets exceedingMaxBucket(defaulting to 5000) and processes the remaining candidates. This ensures that most valid type-3 clones are still detected, even in very common mini-window patterns. - Added logging for truncation and progress: A warning log is now emitted when a fuzzy bucket is truncated, informing the user that some potential near-miss clones might be skipped. Additionally, info-level progress logs are added for evaluating fuzzy candidate pairs, which can be helpful for large codebases.
- CLI flag and configuration support: The
max-bucketflag has been added to thescancommand, allowing users to specify this option via the CLI. It will also be configurable via the.dupehound.ymlfile. - New test case: A dedicated test
TestDetect_Type3_LargeBucketNotSkippedhas been added to verify that large buckets are now correctly processed and do not prevent type-3 detection.
Concerns
- Potential Performance Degradation: The default
MaxBucketvalue has been increased from an effective 100 (due to the previous hard skip) to 5000. While this fixes a correctness issue, it significantly increases the number of pairwise comparisons in large buckets. A bucket of 5000 blocks can lead to(5000 * 4999) / 2= ~12.5 million comparisons for that single bucket. This is a substantial increase from the ~5,000 comparisons for a bucket of 100 blocks. For very large codebases with highly repetitive patterns, this could lead to noticeable performance degradation, especially if many such large buckets exist. While the option is configurable, the default might be too aggressive for general use without more prominent warnings or guidance on its impact.
Verdict
Request Changes
Code review performed by GEMINI - gemini-2.5-flash.
| @@ -305,13 +326,21 @@ func detectFuzzy(files []TokenizedFile, minTokens int, threshold float64, exactC | |||
| var pairs []pair | |||
There was a problem hiding this comment.
[WARNING] The default maxBucket value has been increased from an effective 100 to 5000. This is a 50x increase in the number of blocks considered, leading to a 50^2 = 2500x increase in worst-case comparisons per bucket. While this fixes a correctness issue, it could lead to significant performance degradation on large codebases with many common mini-hashes. Consider a more conservative default (e.g., 500 or 1000) or add more prominent documentation/warnings about the performance implications of a high maxBucket value.
| @@ -305,13 +326,21 @@ func detectFuzzy(files []TokenizedFile, minTokens int, threshold float64, exactC | |||
| var pairs []pair | |||
There was a problem hiding this comment.
[SUGGESTION] The warning message for fuzzy bucket truncated is at Warn level. Given the potential performance impact of a high maxBucket and the fact that truncation means some clones might be missed, it might be more appropriate to log this at Info or Debug level by default, and only elevate to Warn if the truncation is very severe (e.g., more than 90% of candidates skipped). The current Warn level might be too noisy for normal operation if truncation is common but expected.
| bucket = bucket[:maxBucket] | ||
| } | ||
| for i := 0; i < len(bucket); i++ { | ||
| for j := i + 1; j < len(bucket); j++ { |
There was a problem hiding this comment.
[SUGGESTION] The progress logging for fuzzy candidate pairs is a good addition. However, the threshold for logging (len(pairs) > 10000) and the interval (i % 10000 == 0) might be too high for smaller to medium-sized codebases, meaning users might not see any progress updates even when the fuzzy detection takes a noticeable amount of time. Consider making these thresholds configurable or dynamically adjusting them based on the total number of pairs to provide more granular feedback.
- Rewrite text formatter with hotspots section, top clones by impact, type breakdown, and relative paths. Full clone list preserved for machine/AI consumption. - Add --format md for GitHub PR comments with expandable <details> sections, syntax-highlighted previews, and hotspot tables. - Deduplicate overlapping clones in post-detection step, removing clones fully contained within larger ones. - Add dupehound self-scan job to PR workflow. - Fix flag defaults (min-tokens, similarity, max-bucket) not applied when no config file exists and flags are not explicitly passed.
There was a problem hiding this comment.
Core Changes
This PR introduces significant enhancements to dupehound, focusing on improving the accuracy of type-3 clone detection, providing more actionable and user-friendly output formats, and integrating the tool into the project's CI/CD pipeline.
Key changes include:
- Fuzzy Bucket Handling: The
detectorservice now processes large fuzzy hash buckets more efficiently by truncating them to a configurable maximum size (--max-bucketflag, default 5000), preventing performance degradation on common mini-window patterns. Progress logging has been added for long-running fuzzy detection. - Enhanced Output Formats: The
reporterservice has been refactored to provide a much more informativetextoutput, including hotspots (files with most duplication), clones sorted by impact, and a type breakdown. A newmarkdownformat (--format md) is introduced, specifically designed for GitHub PR comments, utilizing<details>tags for expandability and syntax-highlighted code previews. - Overlapping Clone Deduplication: A post-detection step in the
scannerservice (deduplicateOverlapping) now removes smaller clones that are fully contained within larger, already reported clones. This significantly reduces noise in the report (claimed 22% reduction on real codebases). - GitHub Actions Integration: A new
dupehoundjob has been added to the PR workflow (pr.yml) to self-scan the codebase and post the markdown report as a PR comment, automatically updating previous comments. - Bug Fix: Corrected the application of default values for
min-tokens,similarity, andmax-bucketto ensure CLI flags, config file settings, and built-in defaults are applied in the correct precedence.
Concerns
- Go Version in Workflow: The specified Go version
1.24in the GitHub Actions workflow is not yet released, which will cause the workflow to fail. - Flag Default Application Logic: There's a potential bug in how CLI flag defaults are re-applied after config file defaults, which could inadvertently override config-set values with flag defaults if the flag wasn't explicitly provided.
Verdict
Request Changes
Code review performed by GEMINI - gemini-2.5-flash.
| config: .structlint.yaml | ||
| comment-on-pr: "true" | ||
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
|
|
There was a problem hiding this comment.
[CRITICAL] The go-version: '1.24' specified here is for an unreleased version of Go. This will cause the workflow to fail. Please update this to a currently stable and released version, such as 1.22.
| if c.IsSet("max-bucket") { | ||
| opts.MaxBucket = c.Int("max-bucket") | ||
| } | ||
|
|
There was a problem hiding this comment.
[WARNING] The logic for applying default values for MinTokens, MinSimilarity, and MaxBucket after services.ApplyConfigDefaults is potentially problematic. If a value is set in the config file but not explicitly via a CLI flag, ApplyConfigDefaults will set opts.X to the config value. However, c.Int("min-tokens") (or c.Float64("similarity"), c.Int("max-bucket")) will return the flag's default value if the flag was not explicitly set, potentially overriding the config value with the flag's default.
To fix this, these if opts.X == 0 checks should only use the built-in default if neither CLI nor config provided a value. A safer approach would be to initialize opts with built-in defaults, then apply config, then apply CLI flags (which cli.Context already handles if c.IsSet is checked). Alternatively, ensure ApplyConfigDefaults correctly handles zero values from CLI flags that were not set, and then these if blocks would only apply the true built-in defaults if opts.X is still zero.
| } | ||
| clones := Detect(tokenizedFiles, minTokens, minSimilarity) | ||
| clones := DetectWithOptions(tokenizedFiles, DetectOptions{ | ||
| MinTokens: minTokens, |
There was a problem hiding this comment.
[SUGGESTION] The deduplicateOverlapping function is a valuable addition for reducing noise in the reports. Currently, it removes clones only if their instances are fully contained within a larger clone's instances. This is a valid design choice, but it means partial overlaps will still result in both clones being reported. Consider if a more aggressive deduplication strategy (e.g., if any significant overlap exists) might be beneficial, or if the current behavior is explicitly intended and should be documented.
There was a problem hiding this comment.
Core Changes
This PR introduces significant enhancements to dupehound, focusing on improving the accuracy of type-3 clone detection, providing more actionable and user-friendly output formats, and integrating the tool into the project's CI/CD pipeline.
Key changes include:
- Fuzzy Bucket Handling: The
detectorservice now processes large fuzzy hash buckets more efficiently by truncating them to a configurable maximum size (--max-bucketflag, default 5000), preventing performance degradation on common mini-window patterns. Progress logging has been added for long-running fuzzy detection. - Enhanced Output Formats: The
reporterservice has been refactored to provide a much more informativetextoutput, including hotspots (files with most duplication), clones sorted by impact, and a type breakdown. A newmarkdownformat (--format md) is introduced, specifically designed for GitHub PR comments, utilizing<details>tags for expandability and syntax-highlighted code previews. - Overlapping Clone Deduplication: A post-detection step in the
scannerservice (deduplicateOverlapping) now removes smaller clones that are fully contained within larger, already reported clones. This significantly reduces noise in the report (claimed 22% reduction on real codebases). - GitHub Actions Integration: A new
dupehoundjob has been added to the PR workflow (pr.yml) to self-scan the codebase and post the markdown report as a PR comment, automatically updating previous comments. - Bug Fix: Corrected the application of default values for
min-tokens,similarity, andmax-bucketto ensure CLI flags, config file settings, and built-in defaults are applied in the correct precedence.
Concerns
- Go Version in Workflow: The specified Go version
1.24in the GitHub Actions workflow is not yet released, which will cause the workflow to fail. - Flag Default Application Logic: There's a potential bug in how CLI flag defaults are re-applied after config file defaults, which could inadvertently override config-set values with flag defaults if the flag wasn't explicitly provided.
Verdict
Request Changes
Code review performed by GEMINI - gemini-2.5-flash.
| config: .structlint.yaml | ||
| comment-on-pr: "true" | ||
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
|
|
There was a problem hiding this comment.
[CRITICAL] The go-version: '1.24' specified here is for an unreleased version of Go. This will cause the workflow to fail. Please update this to a currently stable and released version, such as 1.22.
| if c.IsSet("max-bucket") { | ||
| opts.MaxBucket = c.Int("max-bucket") | ||
| } | ||
|
|
There was a problem hiding this comment.
[WARNING] The logic for applying default values for MinTokens, MinSimilarity, and MaxBucket after services.ApplyConfigDefaults is potentially problematic. If a value is set in the config file but not explicitly via a CLI flag, ApplyConfigDefaults will set opts.X to the config value. However, c.Int("min-tokens") (or c.Float64("similarity"), c.Int("max-bucket")) will return the flag's default value if the flag was not explicitly set, potentially overriding the config value with the flag's default.
To fix this, these if opts.X == 0 checks should only use the built-in default if neither CLI nor config provided a value. A safer approach would be to initialize opts with built-in defaults, then apply config, then apply CLI flags (which cli.Context already handles if c.IsSet is checked). Alternatively, ensure ApplyConfigDefaults correctly handles zero values from CLI flags that were not set, and then these if blocks would only apply the true built-in defaults if opts.X is still zero.
| } | ||
| clones := Detect(tokenizedFiles, minTokens, minSimilarity) | ||
| clones := DetectWithOptions(tokenizedFiles, DetectOptions{ | ||
| MinTokens: minTokens, |
There was a problem hiding this comment.
[SUGGESTION] The deduplicateOverlapping function is a valuable addition for reducing noise in the reports. Currently, it removes clones only if their instances are fully contained within a larger clone's instances. This is a valid design choice, but it means partial overlaps will still result in both clones being reported. Consider if a more aggressive deduplication strategy (e.g., if any significant overlap exists) might be beneficial, or if the current behavior is explicitly intended and should be documented.
…nfig - Markdown format shows top 10 clones instead of dumping all (PR comments were too noisy with 136 expandables). - Workflow appends run URL footer for full results. - Add .dupehound.yml excluding test files from self-scan.
There was a problem hiding this comment.
Core Changes
This PR introduces significant enhancements to dupehound, focusing on improved clone detection, more actionable reporting, and better integration with CI/CD workflows.
Key changes include:
- Fuzzy Clone Detection Refinement: The
detectFuzzyalgorithm now processes larger mini-hash buckets, preventing silent skipping of potential type-3 clones. A new--max-bucketflag and configuration option allow users to control the trade-off between thoroughness and performance. Progress logging has been added for long-running fuzzy detection. - Enhanced Output Formats: A new
md(Markdown) output format is introduced, specifically designed for GitHub PR comments. This format usesdetailstags for expandability, syntax highlighting for code previews, and presents results in a more digestible way with hotspots and top clones. The existingtextformat has also been improved with hotspots, clone type breakdowns, and sorting by impact. - Overlapping Clone Deduplication: A post-detection step (
deduplicateOverlapping) has been added to remove smaller clone instances that are fully contained within larger, already reported clones. This reduces noise and improves the clarity of the reported findings. - GitHub Actions Integration: A new
dupehoundjob is added to the PR workflow, which self-scans the codebase and posts a formatted Markdown report as a PR comment, automatically updating or deleting previous comments. - Configuration Handling Fix: Corrected a bug where CLI flag defaults for
min-tokens,similarity, andmax-bucketwere not correctly applied if no configuration file was present.
Concerns
- Performance: While the
max-bucketflag and logging help manage the performance impact of processing larger fuzzy buckets, the removal of thelen(indices) > 100skip indetectFuzzycould still lead to performance degradation ifmax-bucketis set very high or to 0 (unlimited) on extremely large codebases with many common mini-window patterns. ThededuplicateOverlappingfunction also adds a sorting and iteration step that could beO(N*M)in the worst case forNclones andMcovered ranges, potentially impacting performance for very large reports. - CI Error Visibility: The
2>/dev/nullin the GitHub Actions workflow fordupehound scanmight suppress important error messages from the tool itself, making debugging harder if the scan fails due to an internal error rather than finding clones.
Verdict
Request Changes
Code review performed by GEMINI - gemini-2.5-flash.
There was a problem hiding this comment.
Core Changes
This PR introduces several significant enhancements to dupehound:
- Fuzzy Bucket Handling: The clone detection now properly processes large fuzzy hash buckets, preventing silent skipping of potential type-3 clones. A new
--max-bucketflag (default 5000) allows configuration, and progress/warning logs are added for visibility. - Improved Output Formats: The
textoutput is enhanced with a 'Hotspots' section (files with most duplication), clones sorted by impact, and a breakdown of clone types. A newmd(Markdown) format is added, designed for GitHub PR comments, featuring collapsible<details>sections, syntax-highlighted code previews, and tables. - Overlapping Clone Deduplication: A post-detection step (
deduplicateOverlapping) is introduced to remove smaller clones that are fully contained within larger, already reported clones. This reduces noise in the report by ~22% on tested codebases. - GitHub Actions Integration: A new workflow job is added to automatically run dupehound on PRs and post a formatted Markdown report as a PR comment.
- Flag Default Fixes: Corrected logic for applying default values for
min-tokens,similarity, andmax-bucketwhen no config file is present.
Concerns
- Logic for applying defaults: The logic for applying default values for
min-tokens,min-similarity, andmax-bucketinactions/scan.goappears redundant and potentially confusing. Thecli.Contextmethods likec.Int("flag")already return the default value if the flag is not explicitly set. The currentif opts.MinTokens <= 0checks, afterApplyConfigDefaultsandc.IsSetchecks, might lead to unexpected overrides or unnecessary complexity. - Lack of
deduplicateOverlappingtests: ThededuplicateOverlappingfunction inservices/scanner.gois a new and critical piece of logic. It lacks dedicated unit tests to ensure its correctness across various overlapping scenarios (e.g., partial overlaps, exact overlaps, multiple small clones within a large one, clones in different files). - Performance of
deduplicateOverlapping: For extremely large codebases with many clones, the nested loop structure withindeduplicateOverlapping(iteratingcoveredRangesfor each instance of each clone) could potentially lead to performance degradation. WhilecoveredRangesis a slice, a map-based approach for faster lookup might be considered if performance becomes an issue. - Inconsistent
maxBucketdefault: ThemaxBucketdefault is hardcoded to5000inservices/detector.go'sDetectWithOptionsfunction, butservices/scanner.gopassesopts.MaxBucketwhich has a default of0inScanOptions. This inconsistency means the5000default is applied indetector.goifopts.MaxBucketis0, which is likely intended, but the flow could be clearer by centralizing the default application.
Verdict
Request Changes
Code review performed by GEMINI - gemini-2.5-flash.
| - uses: actions/setup-go@v5 | ||
| with: | ||
| go-version: '1.24' | ||
| cache: true |
There was a problem hiding this comment.
[SUGGESTION] The --verbose flag is typically used to output more detailed logs to stderr. However, 2>/dev/null redirects stderr to /dev/null, effectively discarding any verbose output. If the intention is to capture verbose logs for debugging, consider redirecting stderr to a file or to stdout (2>&1) and then processing it, or remove --verbose if it's not needed for this specific command's purpose (which seems to be just checking the exit code).
| with: | ||
| go-version: '1.24' | ||
| cache: true | ||
| - name: Build dupehound |
There was a problem hiding this comment.
[SUGGESTION] Similar to the previous comment, redirecting stderr to /dev/null here means any errors during the Markdown report generation will not be visible in the GitHub Actions logs. It's generally safer to redirect stderr to stdout (2>&1) or a separate log file for better debugging in CI environments.
| if c.IsSet("max-bucket") { | ||
| opts.MaxBucket = c.Int("max-bucket") | ||
| } | ||
|
|
There was a problem hiding this comment.
[WARNING] This block of 'Final fallback' logic seems redundant and potentially confusing given that ApplyConfigDefaults is called earlier and cli.Context methods like c.Int("min-tokens") already return the default value if the flag is not set.
For example, opts.MinTokens = c.Int("min-tokens") would already set opts.MinTokens to 50 if the flag wasn't provided. If opts.MinTokens was set to 0 by the config file, this line would then override it with the CLI default. This might be the intended behavior, but it's not explicitly clear. Consider simplifying this by relying on ApplyConfigDefaults to set defaults from config, and then only applying CLI flags if they were explicitly set (using c.IsSet) to override config or built-in defaults. The current structure can lead to c.Int being called multiple times or overriding a config value of 0 with a CLI default.
| // DetectWithOptions finds all clone groups with full control over detection parameters. | ||
| func DetectWithOptions(files []TokenizedFile, opts DetectOptions) []domain.Clone { | ||
| if len(files) == 0 || opts.MinTokens <= 0 { | ||
| return nil |
There was a problem hiding this comment.
[SUGGESTION] The maxBucket default is hardcoded here to 5000. However, ScanOptions in services/scanner.go also has a MaxBucket field which defaults to 0. The ScanAction correctly sets opts.MaxBucket from the CLI flag. To maintain consistency and centralize default handling, DetectWithOptions should ideally just use opts.MaxBucket directly, and the default value (e.g., 5000) should be applied either in ScanAction (if not set by CLI or config) or within ApplyConfigDefaults in services/scanner.go.
| ) | ||
|
|
||
| // FormatOptions controls text output presentation. | ||
| type FormatOptions struct { |
There was a problem hiding this comment.
[SUGGESTION] The previewMaxLines constant is set to 5. However, in writeCloneMd (line 240), it's multiplied by 2 (previewMaxLines * 2) for markdown previews. This inconsistency might be confusing. Consider making this a configurable option in FormatOptions or using a single, consistent value, or at least documenting why markdown gets more lines by default.
There was a problem hiding this comment.
Core Changes
This PR introduces significant improvements to dupehound's clone detection and reporting capabilities:
- Fuzzy Bucket Handling: The type-3 clone detection now handles large fuzzy hash buckets more robustly, preventing silent skipping of potential near-miss clones. A new
--max-bucketflag (default 5000) allows configuration, and progress/warning logs are added for large bucket processing. - Enhanced Output Formats: The
textand newmd(Markdown) output formats have been greatly improved. They now include a "Hotspots" section (files with most duplication), clones sorted by impact, and a breakdown of clone types. The Markdown format is specifically designed for GitHub PR comments, using<details>tags for expandability and syntax-highlighted code previews. - Overlapping Clone Deduplication: A post-detection step (
deduplicateOverlapping) has been added to remove smaller clones that are fully contained within larger, already reported clones. This significantly reduces noise and improves the clarity of the reported results. - GitHub Actions Integration: A new CI job has been added to the PR workflow that runs dupehound on its own codebase and posts a formatted Markdown report as a PR comment, including logic to update/delete previous comments.
- Flag Default Bug Fix: Corrected the logic for applying default values for
min-tokens,similarity, andmax-bucketwhen no configuration file is present or flags are not explicitly set.
Concerns
- Performance of Fuzzy Detection: While the
max-bucketparameter and logging are good additions, the fuzzy detection process, especially withmaxBucket = 0(unlimited), can still be computationally intensive on very large codebases. The warning log for truncated buckets is helpful, but users should be aware of the potential performance trade-off when adjusting this parameter. - Redundant Default Logic: The logic for applying default values in
actions/scan.goforMinTokens,MinSimilarity, andMaxBucketappears to be partially redundant with theservices.ApplyConfigDefaultscall and the default values set inflags.go. While it might function correctly due tourfave/cli's behavior, it could be simplified for clarity and maintainability.
Verdict
Comment
Code review performed by GEMINI - gemini-2.5-flash.
| go-version: '1.24' | ||
| cache: true | ||
| - name: Build dupehound | ||
| run: make build-local |
There was a problem hiding this comment.
[SUGGESTION] The 2>/dev/null for the verbose scan will suppress all stderr output, including the verbose logs that --verbose is intended to produce. If the goal is to see verbose output in the CI logs, this redirection should be removed or adjusted (e.g., 1>/dev/null if only stdout is to be suppressed). Given the PR description mentions "progress logging", suppressing stderr seems counterproductive for debugging or understanding the verbose run.
| if c.IsSet("max-bucket") { | ||
| opts.MaxBucket = c.Int("max-bucket") | ||
| } | ||
|
|
There was a problem hiding this comment.
[SUGGESTION] The if opts.MinTokens <= 0 { opts.MinTokens = c.Int("min-tokens") } block is redundant. The c.Int("min-tokens") call will already return the default value (50) if the flag was not explicitly set. If opts.MinTokens is still 0 after ApplyConfigDefaults, it means neither CLI nor config set it, and c.Int("min-tokens") will correctly provide the CLI default. The subsequent if opts.MinTokens <= 0 { return helpers.ErrInvalidMinTokens } check is also redundant because minTokens is guaranteed to be >= 50 at this point due to the line above it, or it would have been set by ApplyConfigDefaults or c.IsSet.
|
|
||
| services.ApplyConfigDefaults(&opts, cfg) | ||
|
|
||
| // Final fallback: nothing set in CLI or config → use built-in defaults. |
There was a problem hiding this comment.
[SUGGESTION] Similar to MinTokens, this fallback logic for MinSimilarity and MaxBucket is redundant. c.Float64("similarity") and c.Int("max-bucket") will already return their default values (0.70 and 5000 respectively) if the flags were not explicitly set. The ApplyConfigDefaults function should ideally handle filling these from the config, and if still zero, the CLI defaults are already available via c.Float64 and c.Int without explicit if opts.Field == 0 checks here.
| @@ -305,13 +326,21 @@ func detectFuzzy(files []TokenizedFile, minTokens int, threshold float64, exactC | |||
| var pairs []pair | |||
There was a problem hiding this comment.
[SUGGESTION] The previous condition len(indices) > 100 was removed, which is good as it prevented detection in large buckets. The new maxBucket logic correctly truncates the bucket if it's too large, with a warning log. This is a significant improvement for type-3 detection accuracy in dense code.
| bucket = bucket[:maxBucket] | ||
| } | ||
| for i := 0; i < len(bucket); i++ { | ||
| for j := i + 1; j < len(bucket); j++ { |
There was a problem hiding this comment.
[SUGGESTION] Adding progress logging for fuzzy candidate pair evaluation is a great improvement for user experience, especially on large codebases where this step can be time-consuming.
There was a problem hiding this comment.
Core Changes
This Pull Request introduces significant enhancements to dupehound, primarily focusing on improving the accuracy of type-3 (near-miss) clone detection and vastly improving the user experience through enhanced reporting capabilities.
Key changes include:
- Improved Type-3 Clone Detection: The fuzzy detection algorithm (
services/detector.go) no longer silently skips large buckets of mini-hashes. Instead, it now processes them with a configurable--max-bucketlimit (defaulting to 5000), significantly reducing false negatives for near-miss clones. Progress logging is added for large fuzzy scans. - Enhanced Text and Markdown Output: The
services/reporter.gopackage has been refactored to provide more actionable insights. The new text format includes a breakdown of clone types, a "Hotspots" section highlighting files with the most duplication, and clones sorted by impact. A new--format mdoption generates GitHub PR comment-ready markdown, featuring collapsible<details>sections, tables, and syntax-highlighted code previews. - Overlapping Clone Deduplication: A post-detection step (
services/scanner.go) now removes smaller clones that are fully contained within larger, already reported clones. This reduces noise in the report by approximately 22% on tested codebases. - PR Workflow Integration: A new GitHub Actions job (
.github/workflows/pr.yml) has been added to automatically run dupehound on pull requests and post a markdown report as a comment, deleting previous dupehound comments to keep the PR clean. - Bug Fix: Default values for
min-tokens,similarity, andmax-bucketare now correctly applied when no configuration file is present, addressing a previous bug.
Concerns
No critical concerns were identified. The changes improve correctness and usability without introducing obvious performance regressions or security vulnerabilities. The max-bucket change is well-handled with logging and configurability.
Verdict
Approve
Code review performed by GEMINI - gemini-2.5-flash.
Restrict clone detection to function/method bodies only, filtering out data declarations, imports, struct definitions, config blocks, and other top-level boilerplate that produces false positives. Three strategies by language family: - Brace + func keyword (Go, JS, TS, Rust, Swift, Kotlin, PHP, etc.) - Brace + depth heuristic (Java, C, C++, C#, Dart) - def/end tracking (Ruby, Elixir) - def/colon heuristic (Python) Tested on simplycodes-web: 1136 clones → 97 (92% noise reduction). Adds 11 new tests covering function boundary detection across Go, JavaScript, TypeScript, Python, Rust, Java, and Ruby.
There was a problem hiding this comment.
Core Changes
This Pull Request significantly enhances dupehound's clone detection capabilities and reporting features. Key changes include:
- Function-level Granularity: Clone detection is now restricted to function/method bodies, ignoring top-level declarations, imports, and other boilerplate. This reduces noise and focuses on actionable logic duplication.
- Improved Fuzzy Detection: The type-3 (near-miss) clone detection now handles large fuzzy hash buckets more robustly by introducing a configurable
max-bucketlimit (default 5000) instead of silently skipping them. Progress logging is added for long-running fuzzy evaluations. - Actionable Reporting: The text and new Markdown (
md) output formats have been revamped to provide more actionable insights. This includes:- A "Hotspots" section listing files with the most duplication.
- Clones sorted by impact (line count * instances).
- A breakdown of clone types (type-1, type-2, type-3).
- Relative file paths for better readability.
- Markdown Output for PRs: A new
mdoutput format is introduced, specifically designed for GitHub PR comments. It uses<details>tags for collapsible clone details and includes syntax-highlighted code previews. - Overlapping Clone Deduplication: A post-detection step removes smaller clones that are fully contained within larger, already reported clones. This significantly reduces report noise (22% reduction reported).
- CI Integration: A new GitHub Actions workflow (
pr.yml) is added to run dupehound on PRs, generate a markdown report, and post it as a PR comment. - Default Value Fixes: Correctly applies default values for
min-tokens,similarity, andmax-bucketwhen no configuration file is present.
Concerns
- The
deduplicateOverlappingfunction, while effective, currently uses a linear scan (for _, r := range coveredRanges) within a loop over clones and their instances. For very large codebases with many overlapping clones, this could lead toO((C*I)^2)complexity in the worst case, whereCis the number of clones andIis the average instances per clone. This might become a performance bottleneck.
Verdict
Approve
Code review performed by GEMINI - gemini-2.5-flash.
| config: .structlint.yaml | ||
| comment-on-pr: "true" | ||
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
|
|
There was a problem hiding this comment.
[SUGGESTION] The go-version 1.24 is not yet a stable release. It's generally recommended to use the latest stable version, which is 1.22 or 1.23 at the time of this review. Using a future version might lead to unexpected build failures if there are breaking changes or if the runner environment doesn't support it yet.
| @@ -183,6 +190,51 @@ func DetectLanguage(path string) *domain.Language { | |||
| return nil | |||
There was a problem hiding this comment.
[WARNING] The deduplicateOverlapping function's performance could be improved. The inner loop for _, r := range coveredRanges performs a linear scan for each instance. If coveredRanges grows very large (e.g., many clones with many instances), this could become a performance bottleneck. Consider optimizing coveredRanges to allow for faster lookups, perhaps by using a map[string][]lineRange where the inner []lineRange is sorted, enabling binary search for overlap checks.
Add dedicated tests for SQL (BEGIN/END) and Lua (function/end) handlers. Cover Go, Python, JavaScript, TypeScript, Java, Kotlin, Rust, C, C++, C#, Swift, Scala, PHP, Ruby, Shell, SQL, Lua, Elixir, Dart, R, and nil-lang fallback.
dupehound scan results
Hotspots
Top clones (by impact)#1
|
There was a problem hiding this comment.
Core Changes
This PR introduces significant enhancements to dupehound, focusing on improving the accuracy, performance, and usability of clone detection and reporting.
Key changes include:
- Function-level granularity: The core clone detection logic (
services/detector.go) now includes aInFuncboolean slice per token, indicating whether a token is part of a function or method body. This allows the detector to filter out boilerplate code (e.g., imports, global variables, struct definitions) and focus on actual executable logic, reducing false positives and improving relevance. This is supported by new language-specific heuristics inservices/tokenizer.go. - Fuzzy bucket handling: The type-3 (near-miss) clone detection now includes a
max-bucketoption (--max-bucketCLI flag, default 5000) to limit the number of candidates processed per fuzzy hash bucket. This preventsO(N^2)performance degradation on very common mini-window patterns, with logging to indicate when truncation occurs. - Actionable reporting: The
services/reporter.gopackage has been refactored to provide more insightful output. This includes:- A "Hotspots" section listing files with the most duplication.
- Clones sorted by impact (line count * instances).
- A breakdown of clone types (type-1, type-2, type-3).
- Relative file paths in reports.
- Markdown output format: A new
--format mdoption generates GitHub PR comment-ready markdown, utilizing<details>tags for expandability and syntax-highlighted code previews. This is integrated into a new GitHub Actions workflow. - Overlapping clone deduplication: A post-detection step (
services/scanner.go) removes smaller clones that are fully contained within larger, already reported clones. This reduces noise and redundancy in the final report, improving clarity. - PR workflow integration: A new GitHub Actions job (
.github/workflows/pr.yml) automatically runs dupehound on pull requests and posts the markdown report as a PR comment.
Concerns
- Python function boundary heuristic: The
markPythonFunctionslogic inservices/tokenizer.gorelies on a heuristic for indentation (checking for blank line gaps ordef/classkeywords on lines with similar indentation). While a reasonable compromise given the token stream's lack of explicit indentation information, it might not be perfectly accurate for all Python code structures, potentially leading to minor inaccuracies in function body detection in edge cases. deduplicateOverlappingperformance: ThededuplicateOverlappingfunction inservices/scanner.goiterates through all clones and their instances, checking against a growing list ofcoveredRanges. In a worst-case scenario with many overlapping clones, this could lead toO(C^2 * I^2)complexity, where C is the number of clones and I is the average instances per clone. For very large codebases with extensive duplication, this might introduce a noticeable performance overhead, although for typical use cases, it should be acceptable.
Verdict
Approve
Code review performed by GEMINI - gemini-2.5-flash.
Summary
--max-bucketflag (default 5000) and progress logging--format md): GitHub PR comment ready with<details>expandables, syntax-highlighted previews, hotspot tablesmin-tokens,similarity,max-bucket) now applied correctly when no config file existsTest plan
<details>tags