Skip to content

feat: fuzzy bucket handling, actionable output, and markdown format - #11

Merged
machado144 merged 10 commits into
mainfrom
fix/fuzzy-no-skip-large-buckets
Apr 2, 2026
Merged

machado144 merged 10 commits into
mainfrom
fix/fuzzy-no-skip-large-buckets

Conversation

@machado144

@machado144 machado144 commented Apr 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Fuzzy bucket handling: process large fuzzy buckets instead of silently skipping them, with configurable --max-bucket flag (default 5000) and progress logging
  • Actionable text output: hotspots section (top files by duplication), clones sorted by impact, type breakdown, relative paths. Full clone list preserved for machine/AI piping
  • Markdown format (--format md): GitHub PR comment ready with <details> expandables, syntax-highlighted previews, hotspot tables
  • Overlapping clone dedup: post-detection step removes clones fully contained within larger ones (22% noise reduction on real codebases)
  • PR workflow: dupehound self-scan job that posts markdown report as PR comment
  • Bug fix: flag defaults (min-tokens, similarity, max-bucket) now applied correctly when no config file exists

Test plan

  • 126 unit + integration tests pass
  • Tested on simplycodes-web (306 files, 883 clones) — no crash, ~5s, ~2MB RSS
  • Text output: summary fits in first 25 lines, full list preserved below
  • Markdown output: renders correctly with GitHub <details> tags
  • Self-scan: dupehound reports on its own codebase cleanly
  • PR workflow: this PR is the first live test of the dupehound comment job

Replace the hard >100 skip with a cap at 500 blocks per mini-hash bucket.
Blocks beyond the cap are truncated (not skipped entirely), and a debug
log message is emitted so users know when it happens. This ensures valid
type-3 clones in moderately large buckets are still detected.
@github-actions

github-actions Bot commented Apr 2, 2026 •

Copy link
Copy Markdown

StructLint — All checks passed

48 rules validated against .structlint.yaml. No violations found.

View full run · Powered by StructLint

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This pull request addresses a limitation in the detectFuzzy function where mini-hash buckets exceeding 100 entries were entirely skipped, potentially leading to missed type-3 (near-miss) clones in repetitive codebases. The change introduces a maxBucketSize constant (set to 500) and modifies the logic to truncate these large buckets to this maximum size instead of skipping them. This ensures that a significant portion of potential clone pairs within large buckets are still considered for Jaccard similarity evaluation. A debug log message is now emitted when a bucket is truncated, providing visibility into this behavior.

Concerns

While the maxBucketSize of 500 is a reasonable heuristic to balance performance and detection completeness, it is still an arbitrary cap. In extremely large buckets (e.g., thousands of entries), it's theoretically possible that the most relevant or highest-similarity clone pairs might exist beyond the first 500 entries and thus be missed. However, this is a necessary trade-off to prevent O(n^2) performance degradation, and the current approach is a significant improvement over skipping these buckets entirely.

Verdict

Approve


Code review performed by GEMINI - gemini-2.5-flash.

Add test verifying large buckets still produce type-3 clones.
Upgrade truncation log from debug to warn so users see it without
--verbose, since it affects result completeness. Improve log message
with actionable context (total, evaluated, skipped counts).

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This Pull Request significantly improves the fuzzy (type-3) clone detection logic in services/detector.go. Previously, mini-hash buckets exceeding 100 blocks were entirely skipped, leading to missed near-miss clones in highly repetitive codebases. This change replaces the hard skip with a truncation mechanism, processing up to maxBucketSize (500) blocks per mini-hash bucket. A warning log is now emitted when a bucket is truncated, providing visibility into potential missed clones due to the performance cap.

Concerns

  • Performance Trade-off: While the maxBucketSize of 500 is a reasonable cap to prevent O(n^2) blowup, processing up to 125,000 pairs for a single mini-hash bucket could still be computationally intensive in extreme cases. The warning log helps identify when this truncation occurs, but users should be aware of this trade-off between completeness and performance for very large, highly repetitive codebases.

Verdict

Approve: The changes are a clear improvement over the previous behavior, addressing a significant limitation in fuzzy clone detection. The introduction of a configurable cap with logging provides a good balance between performance and accuracy. The new test case adequately covers the intended behavior.


Code review performed by GEMINI - gemini-2.5-flash.

…ogging

Bump default fuzzy bucket cap from 500 to 5000 (~12.5M pairs, still fast).
Add --max-bucket flag so users can tune or remove the limit. Add progress
logging: info-level for >1000 pairs, periodic updates every 10k pairs.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This Pull Request addresses a critical issue where type-3 (near-miss) clones were being silently missed in dupehound when mini-hash buckets grew too large. Previously, any bucket with more than 100 blocks was entirely skipped, leading to false negatives in clone detection for repetitive codebases.

The core changes include:

  • Introduced MaxBucket option: A new MaxBucket integer option has been added to ScanOptions and DetectOptions to control the maximum number of candidate blocks processed per fuzzy hash bucket. This allows users to tune the trade-off between thoroughness and performance for type-3 detection.
  • Replaced hard skip with truncation: The detectFuzzy function no longer skips large buckets entirely. Instead, it truncates buckets exceeding MaxBucket (defaulting to 5000) and processes the remaining candidates. This ensures that most valid type-3 clones are still detected, even in very common mini-window patterns.
  • Added logging for truncation and progress: A warning log is now emitted when a fuzzy bucket is truncated, informing the user that some potential near-miss clones might be skipped. Additionally, info-level progress logs are added for evaluating fuzzy candidate pairs, which can be helpful for large codebases.
  • CLI flag and configuration support: The max-bucket flag has been added to the scan command, allowing users to specify this option via the CLI. It will also be configurable via the .dupehound.yml file.
  • New test case: A dedicated test TestDetect_Type3_LargeBucketNotSkipped has been added to verify that large buckets are now correctly processed and do not prevent type-3 detection.

Concerns

  • Potential Performance Degradation: The default MaxBucket value has been increased from an effective 100 (due to the previous hard skip) to 5000. While this fixes a correctness issue, it significantly increases the number of pairwise comparisons in large buckets. A bucket of 5000 blocks can lead to (5000 * 4999) / 2 = ~12.5 million comparisons for that single bucket. This is a substantial increase from the ~5,000 comparisons for a bucket of 100 blocks. For very large codebases with highly repetitive patterns, this could lead to noticeable performance degradation, especially if many such large buckets exist. While the option is configurable, the default might be too aggressive for general use without more prominent warnings or guidance on its impact.

Verdict

Request Changes


Code review performed by GEMINI - gemini-2.5-flash.

Comment thread services/detector.go
@@ -305,13 +326,21 @@ func detectFuzzy(files []TokenizedFile, minTokens int, threshold float64, exactC
var pairs []pair

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[WARNING] The default maxBucket value has been increased from an effective 100 to 5000. This is a 50x increase in the number of blocks considered, leading to a 50^2 = 2500x increase in worst-case comparisons per bucket. While this fixes a correctness issue, it could lead to significant performance degradation on large codebases with many common mini-hashes. Consider a more conservative default (e.g., 500 or 1000) or add more prominent documentation/warnings about the performance implications of a high maxBucket value.

Comment thread services/detector.go
@@ -305,13 +326,21 @@ func detectFuzzy(files []TokenizedFile, minTokens int, threshold float64, exactC
var pairs []pair

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] The warning message for fuzzy bucket truncated is at Warn level. Given the potential performance impact of a high maxBucket and the fact that truncation means some clones might be missed, it might be more appropriate to log this at Info or Debug level by default, and only elevate to Warn if the truncation is very severe (e.g., more than 90% of candidates skipped). The current Warn level might be too noisy for normal operation if truncation is common but expected.

Comment thread services/detector.go
bucket = bucket[:maxBucket]
}
for i := 0; i < len(bucket); i++ {
for j := i + 1; j < len(bucket); j++ {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] The progress logging for fuzzy candidate pairs is a good addition. However, the threshold for logging (len(pairs) > 10000) and the interval (i % 10000 == 0) might be too high for smaller to medium-sized codebases, meaning users might not see any progress updates even when the fuzzy detection takes a noticeable amount of time. Consider making these thresholds configurable or dynamically adjusting them based on the total number of pairs to provide more granular feedback.

- Rewrite text formatter with hotspots section, top clones by impact,
  type breakdown, and relative paths. Full clone list preserved for
  machine/AI consumption.
- Add --format md for GitHub PR comments with expandable <details>
  sections, syntax-highlighted previews, and hotspot tables.
- Deduplicate overlapping clones in post-detection step, removing
  clones fully contained within larger ones.
- Add dupehound self-scan job to PR workflow.
- Fix flag defaults (min-tokens, similarity, max-bucket) not applied
  when no config file exists and flags are not explicitly passed.
@machado144 machado144 changed the title fix: process large fuzzy buckets instead of skipping feat: fuzzy bucket handling, actionable output, and markdown format Apr 2, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This PR introduces significant enhancements to dupehound, focusing on improving the accuracy of type-3 clone detection, providing more actionable and user-friendly output formats, and integrating the tool into the project's CI/CD pipeline.

Key changes include:

  • Fuzzy Bucket Handling: The detector service now processes large fuzzy hash buckets more efficiently by truncating them to a configurable maximum size (--max-bucket flag, default 5000), preventing performance degradation on common mini-window patterns. Progress logging has been added for long-running fuzzy detection.
  • Enhanced Output Formats: The reporter service has been refactored to provide a much more informative text output, including hotspots (files with most duplication), clones sorted by impact, and a type breakdown. A new markdown format (--format md) is introduced, specifically designed for GitHub PR comments, utilizing <details> tags for expandability and syntax-highlighted code previews.
  • Overlapping Clone Deduplication: A post-detection step in the scanner service (deduplicateOverlapping) now removes smaller clones that are fully contained within larger, already reported clones. This significantly reduces noise in the report (claimed 22% reduction on real codebases).
  • GitHub Actions Integration: A new dupehound job has been added to the PR workflow (pr.yml) to self-scan the codebase and post the markdown report as a PR comment, automatically updating previous comments.
  • Bug Fix: Corrected the application of default values for min-tokens, similarity, and max-bucket to ensure CLI flags, config file settings, and built-in defaults are applied in the correct precedence.

Concerns

  • Go Version in Workflow: The specified Go version 1.24 in the GitHub Actions workflow is not yet released, which will cause the workflow to fail.
  • Flag Default Application Logic: There's a potential bug in how CLI flag defaults are re-applied after config file defaults, which could inadvertently override config-set values with flag defaults if the flag wasn't explicitly provided.

Verdict

Request Changes


Code review performed by GEMINI - gemini-2.5-flash.

Comment thread .github/workflows/pr.yml
config: .structlint.yaml
comment-on-pr: "true"
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[CRITICAL] The go-version: '1.24' specified here is for an unreleased version of Go. This will cause the workflow to fail. Please update this to a currently stable and released version, such as 1.22.

Comment thread actions/scan.go
if c.IsSet("max-bucket") {
opts.MaxBucket = c.Int("max-bucket")
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[WARNING] The logic for applying default values for MinTokens, MinSimilarity, and MaxBucket after services.ApplyConfigDefaults is potentially problematic. If a value is set in the config file but not explicitly via a CLI flag, ApplyConfigDefaults will set opts.X to the config value. However, c.Int("min-tokens") (or c.Float64("similarity"), c.Int("max-bucket")) will return the flag's default value if the flag was not explicitly set, potentially overriding the config value with the flag's default.

To fix this, these if opts.X == 0 checks should only use the built-in default if neither CLI nor config provided a value. A safer approach would be to initialize opts with built-in defaults, then apply config, then apply CLI flags (which cli.Context already handles if c.IsSet is checked). Alternatively, ensure ApplyConfigDefaults correctly handles zero values from CLI flags that were not set, and then these if blocks would only apply the true built-in defaults if opts.X is still zero.

Comment thread services/scanner.go
}
clones := Detect(tokenizedFiles, minTokens, minSimilarity)
clones := DetectWithOptions(tokenizedFiles, DetectOptions{
MinTokens: minTokens,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] The deduplicateOverlapping function is a valuable addition for reducing noise in the reports. Currently, it removes clones only if their instances are fully contained within a larger clone's instances. This is a valid design choice, but it means partial overlaps will still result in both clones being reported. Consider if a more aggressive deduplication strategy (e.g., if any significant overlap exists) might be beneficial, or if the current behavior is explicitly intended and should be documented.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This PR introduces significant enhancements to dupehound, focusing on improving the accuracy of type-3 clone detection, providing more actionable and user-friendly output formats, and integrating the tool into the project's CI/CD pipeline.

Key changes include:

  • Fuzzy Bucket Handling: The detector service now processes large fuzzy hash buckets more efficiently by truncating them to a configurable maximum size (--max-bucket flag, default 5000), preventing performance degradation on common mini-window patterns. Progress logging has been added for long-running fuzzy detection.
  • Enhanced Output Formats: The reporter service has been refactored to provide a much more informative text output, including hotspots (files with most duplication), clones sorted by impact, and a type breakdown. A new markdown format (--format md) is introduced, specifically designed for GitHub PR comments, utilizing <details> tags for expandability and syntax-highlighted code previews.
  • Overlapping Clone Deduplication: A post-detection step in the scanner service (deduplicateOverlapping) now removes smaller clones that are fully contained within larger, already reported clones. This significantly reduces noise in the report (claimed 22% reduction on real codebases).
  • GitHub Actions Integration: A new dupehound job has been added to the PR workflow (pr.yml) to self-scan the codebase and post the markdown report as a PR comment, automatically updating previous comments.
  • Bug Fix: Corrected the application of default values for min-tokens, similarity, and max-bucket to ensure CLI flags, config file settings, and built-in defaults are applied in the correct precedence.

Concerns

  • Go Version in Workflow: The specified Go version 1.24 in the GitHub Actions workflow is not yet released, which will cause the workflow to fail.
  • Flag Default Application Logic: There's a potential bug in how CLI flag defaults are re-applied after config file defaults, which could inadvertently override config-set values with flag defaults if the flag wasn't explicitly provided.

Verdict

Request Changes


Code review performed by GEMINI - gemini-2.5-flash.

Comment thread .github/workflows/pr.yml
config: .structlint.yaml
comment-on-pr: "true"
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[CRITICAL] The go-version: '1.24' specified here is for an unreleased version of Go. This will cause the workflow to fail. Please update this to a currently stable and released version, such as 1.22.

Comment thread actions/scan.go
if c.IsSet("max-bucket") {
opts.MaxBucket = c.Int("max-bucket")
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[WARNING] The logic for applying default values for MinTokens, MinSimilarity, and MaxBucket after services.ApplyConfigDefaults is potentially problematic. If a value is set in the config file but not explicitly via a CLI flag, ApplyConfigDefaults will set opts.X to the config value. However, c.Int("min-tokens") (or c.Float64("similarity"), c.Int("max-bucket")) will return the flag's default value if the flag was not explicitly set, potentially overriding the config value with the flag's default.

To fix this, these if opts.X == 0 checks should only use the built-in default if neither CLI nor config provided a value. A safer approach would be to initialize opts with built-in defaults, then apply config, then apply CLI flags (which cli.Context already handles if c.IsSet is checked). Alternatively, ensure ApplyConfigDefaults correctly handles zero values from CLI flags that were not set, and then these if blocks would only apply the true built-in defaults if opts.X is still zero.

Comment thread services/scanner.go
}
clones := Detect(tokenizedFiles, minTokens, minSimilarity)
clones := DetectWithOptions(tokenizedFiles, DetectOptions{
MinTokens: minTokens,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] The deduplicateOverlapping function is a valuable addition for reducing noise in the reports. Currently, it removes clones only if their instances are fully contained within a larger clone's instances. This is a valid design choice, but it means partial overlaps will still result in both clones being reported. Consider if a more aggressive deduplication strategy (e.g., if any significant overlap exists) might be beneficial, or if the current behavior is explicitly intended and should be documented.

…nfig

- Markdown format shows top 10 clones instead of dumping all (PR
  comments were too noisy with 136 expandables).
- Workflow appends run URL footer for full results.
- Add .dupehound.yml excluding test files from self-scan.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This PR introduces significant enhancements to dupehound, focusing on improved clone detection, more actionable reporting, and better integration with CI/CD workflows.

Key changes include:

  • Fuzzy Clone Detection Refinement: The detectFuzzy algorithm now processes larger mini-hash buckets, preventing silent skipping of potential type-3 clones. A new --max-bucket flag and configuration option allow users to control the trade-off between thoroughness and performance. Progress logging has been added for long-running fuzzy detection.
  • Enhanced Output Formats: A new md (Markdown) output format is introduced, specifically designed for GitHub PR comments. This format uses details tags for expandability, syntax highlighting for code previews, and presents results in a more digestible way with hotspots and top clones. The existing text format has also been improved with hotspots, clone type breakdowns, and sorting by impact.
  • Overlapping Clone Deduplication: A post-detection step (deduplicateOverlapping) has been added to remove smaller clone instances that are fully contained within larger, already reported clones. This reduces noise and improves the clarity of the reported findings.
  • GitHub Actions Integration: A new dupehound job is added to the PR workflow, which self-scans the codebase and posts a formatted Markdown report as a PR comment, automatically updating or deleting previous comments.
  • Configuration Handling Fix: Corrected a bug where CLI flag defaults for min-tokens, similarity, and max-bucket were not correctly applied if no configuration file was present.

Concerns

  • Performance: While the max-bucket flag and logging help manage the performance impact of processing larger fuzzy buckets, the removal of the len(indices) > 100 skip in detectFuzzy could still lead to performance degradation if max-bucket is set very high or to 0 (unlimited) on extremely large codebases with many common mini-window patterns. The deduplicateOverlapping function also adds a sorting and iteration step that could be O(N*M) in the worst case for N clones and M covered ranges, potentially impacting performance for very large reports.
  • CI Error Visibility: The 2>/dev/null in the GitHub Actions workflow for dupehound scan might suppress important error messages from the tool itself, making debugging harder if the scan fails due to an internal error rather than finding clones.

Verdict

Request Changes


Code review performed by GEMINI - gemini-2.5-flash.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This PR introduces several significant enhancements to dupehound:

  • Fuzzy Bucket Handling: The clone detection now properly processes large fuzzy hash buckets, preventing silent skipping of potential type-3 clones. A new --max-bucket flag (default 5000) allows configuration, and progress/warning logs are added for visibility.
  • Improved Output Formats: The text output is enhanced with a 'Hotspots' section (files with most duplication), clones sorted by impact, and a breakdown of clone types. A new md (Markdown) format is added, designed for GitHub PR comments, featuring collapsible <details> sections, syntax-highlighted code previews, and tables.
  • Overlapping Clone Deduplication: A post-detection step (deduplicateOverlapping) is introduced to remove smaller clones that are fully contained within larger, already reported clones. This reduces noise in the report by ~22% on tested codebases.
  • GitHub Actions Integration: A new workflow job is added to automatically run dupehound on PRs and post a formatted Markdown report as a PR comment.
  • Flag Default Fixes: Corrected logic for applying default values for min-tokens, similarity, and max-bucket when no config file is present.

Concerns

  • Logic for applying defaults: The logic for applying default values for min-tokens, min-similarity, and max-bucket in actions/scan.go appears redundant and potentially confusing. The cli.Context methods like c.Int("flag") already return the default value if the flag is not explicitly set. The current if opts.MinTokens <= 0 checks, after ApplyConfigDefaults and c.IsSet checks, might lead to unexpected overrides or unnecessary complexity.
  • Lack of deduplicateOverlapping tests: The deduplicateOverlapping function in services/scanner.go is a new and critical piece of logic. It lacks dedicated unit tests to ensure its correctness across various overlapping scenarios (e.g., partial overlaps, exact overlaps, multiple small clones within a large one, clones in different files).
  • Performance of deduplicateOverlapping: For extremely large codebases with many clones, the nested loop structure within deduplicateOverlapping (iterating coveredRanges for each instance of each clone) could potentially lead to performance degradation. While coveredRanges is a slice, a map-based approach for faster lookup might be considered if performance becomes an issue.
  • Inconsistent maxBucket default: The maxBucket default is hardcoded to 5000 in services/detector.go's DetectWithOptions function, but services/scanner.go passes opts.MaxBucket which has a default of 0 in ScanOptions. This inconsistency means the 5000 default is applied in detector.go if opts.MaxBucket is 0, which is likely intended, but the flow could be clearer by centralizing the default application.

Verdict

Request Changes


Code review performed by GEMINI - gemini-2.5-flash.

Comment thread .github/workflows/pr.yml
- uses: actions/setup-go@v5
with:
go-version: '1.24'
cache: true

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] The --verbose flag is typically used to output more detailed logs to stderr. However, 2>/dev/null redirects stderr to /dev/null, effectively discarding any verbose output. If the intention is to capture verbose logs for debugging, consider redirecting stderr to a file or to stdout (2>&1) and then processing it, or remove --verbose if it's not needed for this specific command's purpose (which seems to be just checking the exit code).

Comment thread .github/workflows/pr.yml
with:
go-version: '1.24'
cache: true
- name: Build dupehound

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] Similar to the previous comment, redirecting stderr to /dev/null here means any errors during the Markdown report generation will not be visible in the GitHub Actions logs. It's generally safer to redirect stderr to stdout (2>&1) or a separate log file for better debugging in CI environments.

Comment thread actions/scan.go
if c.IsSet("max-bucket") {
opts.MaxBucket = c.Int("max-bucket")
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[WARNING] This block of 'Final fallback' logic seems redundant and potentially confusing given that ApplyConfigDefaults is called earlier and cli.Context methods like c.Int("min-tokens") already return the default value if the flag is not set.

For example, opts.MinTokens = c.Int("min-tokens") would already set opts.MinTokens to 50 if the flag wasn't provided. If opts.MinTokens was set to 0 by the config file, this line would then override it with the CLI default. This might be the intended behavior, but it's not explicitly clear. Consider simplifying this by relying on ApplyConfigDefaults to set defaults from config, and then only applying CLI flags if they were explicitly set (using c.IsSet) to override config or built-in defaults. The current structure can lead to c.Int being called multiple times or overriding a config value of 0 with a CLI default.

Comment thread services/detector.go
// DetectWithOptions finds all clone groups with full control over detection parameters.
func DetectWithOptions(files []TokenizedFile, opts DetectOptions) []domain.Clone {
if len(files) == 0 || opts.MinTokens <= 0 {
return nil

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] The maxBucket default is hardcoded here to 5000. However, ScanOptions in services/scanner.go also has a MaxBucket field which defaults to 0. The ScanAction correctly sets opts.MaxBucket from the CLI flag. To maintain consistency and centralize default handling, DetectWithOptions should ideally just use opts.MaxBucket directly, and the default value (e.g., 5000) should be applied either in ScanAction (if not set by CLI or config) or within ApplyConfigDefaults in services/scanner.go.

Comment thread services/reporter.go
)

// FormatOptions controls text output presentation.
type FormatOptions struct {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] The previewMaxLines constant is set to 5. However, in writeCloneMd (line 240), it's multiplied by 2 (previewMaxLines * 2) for markdown previews. This inconsistency might be confusing. Consider making this a configurable option in FormatOptions or using a single, consistent value, or at least documenting why markdown gets more lines by default.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This PR introduces significant improvements to dupehound's clone detection and reporting capabilities:

  • Fuzzy Bucket Handling: The type-3 clone detection now handles large fuzzy hash buckets more robustly, preventing silent skipping of potential near-miss clones. A new --max-bucket flag (default 5000) allows configuration, and progress/warning logs are added for large bucket processing.
  • Enhanced Output Formats: The text and new md (Markdown) output formats have been greatly improved. They now include a "Hotspots" section (files with most duplication), clones sorted by impact, and a breakdown of clone types. The Markdown format is specifically designed for GitHub PR comments, using <details> tags for expandability and syntax-highlighted code previews.
  • Overlapping Clone Deduplication: A post-detection step (deduplicateOverlapping) has been added to remove smaller clones that are fully contained within larger, already reported clones. This significantly reduces noise and improves the clarity of the reported results.
  • GitHub Actions Integration: A new CI job has been added to the PR workflow that runs dupehound on its own codebase and posts a formatted Markdown report as a PR comment, including logic to update/delete previous comments.
  • Flag Default Bug Fix: Corrected the logic for applying default values for min-tokens, similarity, and max-bucket when no configuration file is present or flags are not explicitly set.

Concerns

  • Performance of Fuzzy Detection: While the max-bucket parameter and logging are good additions, the fuzzy detection process, especially with maxBucket = 0 (unlimited), can still be computationally intensive on very large codebases. The warning log for truncated buckets is helpful, but users should be aware of the potential performance trade-off when adjusting this parameter.
  • Redundant Default Logic: The logic for applying default values in actions/scan.go for MinTokens, MinSimilarity, and MaxBucket appears to be partially redundant with the services.ApplyConfigDefaults call and the default values set in flags.go. While it might function correctly due to urfave/cli's behavior, it could be simplified for clarity and maintainability.

Verdict

Comment


Code review performed by GEMINI - gemini-2.5-flash.

Comment thread .github/workflows/pr.yml
go-version: '1.24'
cache: true
- name: Build dupehound
run: make build-local

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] The 2>/dev/null for the verbose scan will suppress all stderr output, including the verbose logs that --verbose is intended to produce. If the goal is to see verbose output in the CI logs, this redirection should be removed or adjusted (e.g., 1>/dev/null if only stdout is to be suppressed). Given the PR description mentions "progress logging", suppressing stderr seems counterproductive for debugging or understanding the verbose run.

Comment thread actions/scan.go
if c.IsSet("max-bucket") {
opts.MaxBucket = c.Int("max-bucket")
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] The if opts.MinTokens <= 0 { opts.MinTokens = c.Int("min-tokens") } block is redundant. The c.Int("min-tokens") call will already return the default value (50) if the flag was not explicitly set. If opts.MinTokens is still 0 after ApplyConfigDefaults, it means neither CLI nor config set it, and c.Int("min-tokens") will correctly provide the CLI default. The subsequent if opts.MinTokens <= 0 { return helpers.ErrInvalidMinTokens } check is also redundant because minTokens is guaranteed to be >= 50 at this point due to the line above it, or it would have been set by ApplyConfigDefaults or c.IsSet.

Comment thread actions/scan.go

services.ApplyConfigDefaults(&opts, cfg)

// Final fallback: nothing set in CLI or config → use built-in defaults.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] Similar to MinTokens, this fallback logic for MinSimilarity and MaxBucket is redundant. c.Float64("similarity") and c.Int("max-bucket") will already return their default values (0.70 and 5000 respectively) if the flags were not explicitly set. The ApplyConfigDefaults function should ideally handle filling these from the config, and if still zero, the CLI defaults are already available via c.Float64 and c.Int without explicit if opts.Field == 0 checks here.

Comment thread services/detector.go
@@ -305,13 +326,21 @@ func detectFuzzy(files []TokenizedFile, minTokens int, threshold float64, exactC
var pairs []pair

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] The previous condition len(indices) > 100 was removed, which is good as it prevented detection in large buckets. The new maxBucket logic correctly truncates the bucket if it's too large, with a warning log. This is a significant improvement for type-3 detection accuracy in dense code.

Comment thread services/detector.go
bucket = bucket[:maxBucket]
}
for i := 0; i < len(bucket); i++ {
for j := i + 1; j < len(bucket); j++ {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] Adding progress logging for fuzzy candidate pair evaluation is a great improvement for user experience, especially on large codebases where this step can be time-consuming.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This Pull Request introduces significant enhancements to dupehound, primarily focusing on improving the accuracy of type-3 (near-miss) clone detection and vastly improving the user experience through enhanced reporting capabilities.

Key changes include:

  • Improved Type-3 Clone Detection: The fuzzy detection algorithm (services/detector.go) no longer silently skips large buckets of mini-hashes. Instead, it now processes them with a configurable --max-bucket limit (defaulting to 5000), significantly reducing false negatives for near-miss clones. Progress logging is added for large fuzzy scans.
  • Enhanced Text and Markdown Output: The services/reporter.go package has been refactored to provide more actionable insights. The new text format includes a breakdown of clone types, a "Hotspots" section highlighting files with the most duplication, and clones sorted by impact. A new --format md option generates GitHub PR comment-ready markdown, featuring collapsible <details> sections, tables, and syntax-highlighted code previews.
  • Overlapping Clone Deduplication: A post-detection step (services/scanner.go) now removes smaller clones that are fully contained within larger, already reported clones. This reduces noise in the report by approximately 22% on tested codebases.
  • PR Workflow Integration: A new GitHub Actions job (.github/workflows/pr.yml) has been added to automatically run dupehound on pull requests and post a markdown report as a comment, deleting previous dupehound comments to keep the PR clean.
  • Bug Fix: Default values for min-tokens, similarity, and max-bucket are now correctly applied when no configuration file is present, addressing a previous bug.

Concerns

No critical concerns were identified. The changes improve correctness and usability without introducing obvious performance regressions or security vulnerabilities. The max-bucket change is well-handled with logging and configurability.

Verdict

Approve


Code review performed by GEMINI - gemini-2.5-flash.

Restrict clone detection to function/method bodies only, filtering
out data declarations, imports, struct definitions, config blocks,
and other top-level boilerplate that produces false positives.

Three strategies by language family:
- Brace + func keyword (Go, JS, TS, Rust, Swift, Kotlin, PHP, etc.)
- Brace + depth heuristic (Java, C, C++, C#, Dart)
- def/end tracking (Ruby, Elixir)
- def/colon heuristic (Python)

Tested on simplycodes-web: 1136 clones → 97 (92% noise reduction).
Adds 11 new tests covering function boundary detection across Go,
JavaScript, TypeScript, Python, Rust, Java, and Ruby.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This Pull Request significantly enhances dupehound's clone detection capabilities and reporting features. Key changes include:

  • Function-level Granularity: Clone detection is now restricted to function/method bodies, ignoring top-level declarations, imports, and other boilerplate. This reduces noise and focuses on actionable logic duplication.
  • Improved Fuzzy Detection: The type-3 (near-miss) clone detection now handles large fuzzy hash buckets more robustly by introducing a configurable max-bucket limit (default 5000) instead of silently skipping them. Progress logging is added for long-running fuzzy evaluations.
  • Actionable Reporting: The text and new Markdown (md) output formats have been revamped to provide more actionable insights. This includes:
    • A "Hotspots" section listing files with the most duplication.
    • Clones sorted by impact (line count * instances).
    • A breakdown of clone types (type-1, type-2, type-3).
    • Relative file paths for better readability.
  • Markdown Output for PRs: A new md output format is introduced, specifically designed for GitHub PR comments. It uses <details> tags for collapsible clone details and includes syntax-highlighted code previews.
  • Overlapping Clone Deduplication: A post-detection step removes smaller clones that are fully contained within larger, already reported clones. This significantly reduces report noise (22% reduction reported).
  • CI Integration: A new GitHub Actions workflow (pr.yml) is added to run dupehound on PRs, generate a markdown report, and post it as a PR comment.
  • Default Value Fixes: Correctly applies default values for min-tokens, similarity, and max-bucket when no configuration file is present.

Concerns

  • The deduplicateOverlapping function, while effective, currently uses a linear scan (for _, r := range coveredRanges) within a loop over clones and their instances. For very large codebases with many overlapping clones, this could lead to O((C*I)^2) complexity in the worst case, where C is the number of clones and I is the average instances per clone. This might become a performance bottleneck.

Verdict

Approve


Code review performed by GEMINI - gemini-2.5-flash.

Comment thread .github/workflows/pr.yml
config: .structlint.yaml
comment-on-pr: "true"
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] The go-version 1.24 is not yet a stable release. It's generally recommended to use the latest stable version, which is 1.22 or 1.23 at the time of this review. Using a future version might lead to unexpected build failures if there are breaking changes or if the runner environment doesn't support it yet.

Comment thread services/scanner.go
@@ -183,6 +190,51 @@ func DetectLanguage(path string) *domain.Language {
return nil

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[WARNING] The deduplicateOverlapping function's performance could be improved. The inner loop for _, r := range coveredRanges performs a linear scan for each instance. If coveredRanges grows very large (e.g., many clones with many instances), this could become a performance bottleneck. Consider optimizing coveredRanges to allow for faster lookups, perhaps by using a map[string][]lineRange where the inner []lineRange is sorted, enabling binary search for overlap checks.

Add dedicated tests for SQL (BEGIN/END) and Lua (function/end) handlers.
Cover Go, Python, JavaScript, TypeScript, Java, Kotlin, Rust, C, C++,
C#, Swift, Scala, PHP, Ruby, Shell, SQL, Lua, Elixir, Dart, R, and
nil-lang fallback.
@github-actions

github-actions Bot commented Apr 2, 2026

Copy link
Copy Markdown

dupehound scan results

Metric Value
Files scanned 11 / 11
Clones found 7
Duplicate lines 107
Breakdown 1 type-1, 4 type-2, 2 type-3

Hotspots

Dup Lines File Clones
114 services/reporter.go 10
24 services/detector.go 4

Top clones (by impact)

#1   type-2   similarity: 1.00   19 lines   2 instances

Locations:

  • services/reporter.go:117-135
  • services/reporter.go:197-217

Preview:

	fmt.Fprintf(&b, "dupehound scan results\n")
	fmt.Fprintf(&b, "======================\n")
	fmt.Fprintf(&b, "Files scanned : %d / %d\n", report.ScannedFiles, report.TotalFiles)
	fmt.Fprintf(&b, "Clones found  : %d\n", report.TotalClones)
	fmt.Fprintf(&b, "Duplicate lines: %d\n", report.DuplicateLines)

	if report.TotalClones == 0 {
		fmt.Fprintf(&b, "\nNo duplicates found.\n")
		return b.String()
	}
// ... 9 more lines
#2   type-2   similarity: 1.00   13 lines   2 instances

Locations:

  • services/reporter.go:141-153
  • services/reporter.go:233-246

Preview:

		fmt.Fprintf(&b, "  ... %d more files (use --verbose to show all)\n", len(hotspots)-limit)
	}

	sorted := sortClonesByImpact(report.Clones)

	cloneLimit := defaultTopClones
	if opts.Verbose || len(sorted) <= cloneLimit {
		cloneLimit = len(sorted)
	}

// ... 3 more lines
#3   type-3   similarity: 0.79   9 lines   2 instances

Locations:

  • services/reporter.go:128-136
  • services/reporter.go:208-218

Preview:

	fmt.Fprintf(&b, "Breakdown     : %s\n", typeBreakdown(report.Clones))

	hotspots := buildHotspots(report.Clones, opts.ScanPath)
	limit := defaultTopHotspots
	if opts.Verbose || len(hotspots) <= limit {
		limit = len(hotspots)
	}
	fmt.Fprintf(&b, "\nHotspots (files with most duplication):\n")
	for i := 0; i < limit; i++ {
#4   type-2   similarity: 1.00   7 lines   2 instances

Locations:

  • services/reporter.go:135-141
  • services/reporter.go:219-226

Preview:

	fmt.Fprintf(&b, "\nHotspots (files with most duplication):\n")
	for i := 0; i < limit; i++ {
		hs := hotspots[i]
		fmt.Fprintf(&b, "  %5d lines  %-60s (%d clones)\n", hs.dupLines, hs.path, hs.cloneCount)
	}
	if len(hotspots) > limit {
		fmt.Fprintf(&b, "  ... %d more files (use --verbose to show all)\n", len(hotspots)-limit)
#5   type-1   similarity: 1.00   6 lines   2 instances

Locations:

  • services/detector.go:544-549
  • services/detector.go:559-564

Preview:

	h := fnv.New64a()
	for _, t := range tokens {
		_, _ = h.Write([]byte{byte(t.Kind)})
		if t.Text != "" {
			_, _ = h.Write([]byte(t.Text))
		}
#6   type-2   similarity: 1.00   6 lines   2 instances

Locations:

  • services/reporter.go:180-185
  • services/reporter.go:281-286

Preview:

			for _, line := range lines[:limit] {
				fmt.Fprintf(b, "        | %s\n", strings.TrimRight(line, " \t"))
			}
			if len(lines) > previewMaxLines {
				fmt.Fprintf(b, "        ... (%d more lines)\n", len(lines)-previewMaxLines)
			}
#7   type-3   similarity: 0.94   6 lines   2 instances

Locations:

  • services/detector.go:545-550
  • services/detector.go:560-565

Preview:

	for _, t := range tokens {
		_, _ = h.Write([]byte{byte(t.Kind)})
		if t.Text != "" {
			_, _ = h.Write([]byte(t.Text))
		}
		_, _ = h.Write([]byte{0}) // separator

View full run · Scanned by dupehound

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This PR introduces significant enhancements to dupehound, focusing on improving the accuracy, performance, and usability of clone detection and reporting.

Key changes include:

  • Function-level granularity: The core clone detection logic (services/detector.go) now includes a InFunc boolean slice per token, indicating whether a token is part of a function or method body. This allows the detector to filter out boilerplate code (e.g., imports, global variables, struct definitions) and focus on actual executable logic, reducing false positives and improving relevance. This is supported by new language-specific heuristics in services/tokenizer.go.
  • Fuzzy bucket handling: The type-3 (near-miss) clone detection now includes a max-bucket option (--max-bucket CLI flag, default 5000) to limit the number of candidates processed per fuzzy hash bucket. This prevents O(N^2) performance degradation on very common mini-window patterns, with logging to indicate when truncation occurs.
  • Actionable reporting: The services/reporter.go package has been refactored to provide more insightful output. This includes:
    • A "Hotspots" section listing files with the most duplication.
    • Clones sorted by impact (line count * instances).
    • A breakdown of clone types (type-1, type-2, type-3).
    • Relative file paths in reports.
  • Markdown output format: A new --format md option generates GitHub PR comment-ready markdown, utilizing <details> tags for expandability and syntax-highlighted code previews. This is integrated into a new GitHub Actions workflow.
  • Overlapping clone deduplication: A post-detection step (services/scanner.go) removes smaller clones that are fully contained within larger, already reported clones. This reduces noise and redundancy in the final report, improving clarity.
  • PR workflow integration: A new GitHub Actions job (.github/workflows/pr.yml) automatically runs dupehound on pull requests and posts the markdown report as a PR comment.

Concerns

  • Python function boundary heuristic: The markPythonFunctions logic in services/tokenizer.go relies on a heuristic for indentation (checking for blank line gaps or def/class keywords on lines with similar indentation). While a reasonable compromise given the token stream's lack of explicit indentation information, it might not be perfectly accurate for all Python code structures, potentially leading to minor inaccuracies in function body detection in edge cases.
  • deduplicateOverlapping performance: The deduplicateOverlapping function in services/scanner.go iterates through all clones and their instances, checking against a growing list of coveredRanges. In a worst-case scenario with many overlapping clones, this could lead to O(C^2 * I^2) complexity, where C is the number of clones and I is the average instances per clone. For very large codebases with extensive duplication, this might introduce a noticeable performance overhead, although for typical use cases, it should be acceptable.

Verdict

Approve


Code review performed by GEMINI - gemini-2.5-flash.

@machado144
machado144 merged commit 880621a into main Apr 2, 2026
7 checks passed
@machado144
machado144 deleted the fix/fuzzy-no-skip-large-buckets branch April 2, 2026 14:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

⌂ 换网址