Skip to content

feat(#6): per-file duplication stats and global duplication percentage - #12

Merged
machado144 merged 3 commits into
mainfrom
feat/per-file-duplication-stats
Apr 2, 2026
Merged

machado144 merged 3 commits into
mainfrom
feat/per-file-duplication-stats

Conversation

@machado144

Copy link
Copy Markdown
Contributor

Summary

Closes #6

  • Per-file FileStats with total_lines, duplicate_lines, duplication_pct sorted by percentage descending
  • Global total_lines and duplication_pct added to Report
  • New --min-duplication flag: fail (exit 1) when global duplication % exceeds threshold — enables use as a quality gate
  • Text output shows duplication percentage in header and hotspots sorted by dup %
  • Markdown output shows dup % columns in hotspots table
  • JSON output includes all new fields automatically (snake_case tags)

Test plan

  • 149 existing tests pass
  • Self-scan shows 4.7% duplication (2057 lines, 97 duplicate)
  • simplycodes-web: 6.3% (31,457 lines, 1,984 duplicate)
  • --min-duplication 5.0 --exit-zero on simplycodes-web → exit 1
  • --min-duplication 50.0 --exit-zero → exit 0
  • JSON output includes total_lines, duplication_pct, file_stats
  • Integration tests for threshold (to be added in follow-up)

…d flag

- Add TotalLines, DuplicationPct, and FileStats to Report
- FileStats includes per-file total_lines, duplicate_lines, and
  duplication_pct, sorted by percentage descending
- Add --min-duplication flag: fail (exit 1) when global duplication
  percentage exceeds the threshold
- Update text output with percentage and hotspots sorted by dup %
- Update markdown output with dup % columns in hotspots table
- JSON output includes all new fields automatically
@github-actions

github-actions Bot commented Apr 2, 2026 •

Copy link
Copy Markdown

StructLint — All checks passed

48 rules validated against .structlint.yaml. No violations found.

View full run · Powered by StructLint

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This PR introduces significant enhancements to dupehound's reporting capabilities and adds a new quality gate feature:

  1. Per-file Duplication Statistics: A new FileStats struct is added to the domain.Report, providing total_lines, duplicate_lines, and duplication_pct for each scanned file. These statistics are calculated in services/scanner.go and sorted by duplication percentage for hotspot reporting.
  2. Global Duplication Percentage: The domain.Report now includes TotalLines and DuplicationPct fields, summarizing the overall code duplication across the entire scan.
  3. Duplication Threshold Quality Gate: A new --min-duplication CLI flag (and corresponding logic in actions/scan.go) allows users to specify a maximum acceptable global duplication percentage. If the scan's DuplicationPct exceeds this threshold, the tool will exit with code 1 (similar to finding clones), enabling its use in CI/CD pipelines as a quality gate.
  4. Enhanced Reporting: The text and markdown output formats in services/reporter.go are updated to display the new global and per-file duplication metrics, including percentages and improved hotspot tables.

Concerns (if any)

None. The changes are well-implemented and address the stated goals effectively.

Verdict

Approve


Code review performed by GEMINI - gemini-2.5-flash.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This PR introduces significant enhancements to dupehound's reporting and CLI functionality:

  1. Per-file Duplication Statistics: The domain.Report now includes FileStats which provides total_lines, duplicate_lines, and duplication_pct for each scanned file, sorted by duplication percentage. This allows users to identify 'hotspots' of duplication more easily.
  2. Global Duplication Percentage: The overall total_lines and duplication_pct are now calculated and included in the domain.Report and displayed in the text and Markdown output formats.
  3. Duplication Threshold Quality Gate: A new --min-duplication CLI flag has been added. If set to a value greater than 0, the tool will exit with code 1 (indicating failure) if the global duplication percentage exceeds this threshold. This enables dupehound to be used as a quality gate in CI/CD pipelines.
  4. Improved Output Formats: Both text and Markdown reports have been updated to display the new total lines and duplication percentage metrics, and the 'hotspots' section now leverages the FileStats to show per-file duplication percentages.

Concerns

  • Missing Integration Tests for Quality Gate: The PR description explicitly states that integration tests for the new --min-duplication threshold feature are "to be added in follow-up". This is a critical new feature that directly affects the tool's exit behavior and thus its usability as a quality gate. Without dedicated integration tests, there's a risk of regressions or incorrect behavior in real-world scenarios. These tests should ideally be part of this PR to ensure the feature is fully validated before merging.

Verdict

Request Changes


Code review performed by GEMINI - gemini-2.5-flash.

…eport fields

- Threshold exceeds → exit 1
- Threshold below → exit 0
- Threshold disabled (default 0) → exit 0 with --exit-zero
- JSON output includes total_lines, duplication_pct, file_stats
- Text output includes duplication percentage and Total lines
@github-actions

github-actions Bot commented Apr 2, 2026

Copy link
Copy Markdown

dupehound scan results

Metric Value
Files scanned 11 / 11
Total lines 2015
Clones found 6
Duplicate lines 99 (4.9%)
Breakdown 1 type-1, 4 type-2, 1 type-3

Hotspots

Dup % Dup Lines Total Lines File
19.6% 85 434 services/reporter.go
2.4% 14 586 services/detector.go

Top clones (by impact)

#1   type-2   similarity: 1.00   17 lines   2 instances

Locations:

  • services/reporter.go:94-110
  • services/reporter.go:170-188

Preview:

	fmt.Fprintf(&b, "dupehound scan results\n")
	fmt.Fprintf(&b, "======================\n")
	fmt.Fprintf(&b, "Files scanned : %d / %d\n", report.ScannedFiles, report.TotalFiles)
	fmt.Fprintf(&b, "Total lines   : %d\n", report.TotalLines)
	fmt.Fprintf(&b, "Clones found  : %d\n", report.TotalClones)
	fmt.Fprintf(&b, "Duplicate lines: %d (%.1f%%)\n", report.DuplicateLines, report.DuplicationPct)

	if report.TotalClones == 0 {
		fmt.Fprintf(&b, "\nNo duplicates found.\n")
		return b.String()
// ... 7 more lines
#2   type-2   similarity: 1.00   10 lines   2 instances

Locations:

  • services/reporter.go:117-126
  • services/reporter.go:206-216

Preview:

			fmt.Fprintf(&b, "  ... %d more files (use --verbose to show all)\n", len(report.FileStats)-limit)
		}
	}

	sorted := sortClonesByImpact(report.Clones)
	cloneLimit := topN(len(sorted), defaultTopClones, opts.Verbose)

	fmt.Fprintf(&b, "\nTop clones (by impact):\n")
	for i := 0; i < cloneLimit; i++ {
		writeCloneText(&b, i+1, sorted[i], opts, true)
#3   type-2   similarity: 1.00   9 lines   2 instances

Locations:

  • services/reporter.go:109-117
  • services/reporter.go:189-198

Preview:

		limit := topN(len(report.FileStats), defaultTopHotspots, opts.Verbose)
		fmt.Fprintf(&b, "\nHotspots (files by duplication %%):\n")
		for i := 0; i < limit; i++ {
			fs := report.FileStats[i]
			rel := relPath(fs.File, opts.ScanPath)
			fmt.Fprintf(&b, "  %5.1f%%  %4d/%4d lines  %s\n", fs.DuplicationPct, fs.DuplicateLines, fs.TotalLines, rel)
		}
		if len(report.FileStats) > limit {
			fmt.Fprintf(&b, "  ... %d more files (use --verbose to show all)\n", len(report.FileStats)-limit)
#4   type-1   similarity: 1.00   6 lines   2 instances

Locations:

  • services/detector.go:544-549
  • services/detector.go:559-564

Preview:

	h := fnv.New64a()
	for _, t := range tokens {
		_, _ = h.Write([]byte{byte(t.Kind)})
		if t.Text != "" {
			_, _ = h.Write([]byte(t.Text))
		}
#5   type-2   similarity: 1.00   6 lines   2 instances

Locations:

  • services/reporter.go:153-158
  • services/reporter.go:251-256

Preview:

			for _, line := range lines[:limit] {
				fmt.Fprintf(b, "        | %s\n", strings.TrimRight(line, " \t"))
			}
			if len(lines) > previewMaxLines {
				fmt.Fprintf(b, "        ... (%d more lines)\n", len(lines)-previewMaxLines)
			}
#6   type-3   similarity: 0.94   6 lines   2 instances

Locations:

  • services/detector.go:545-550
  • services/detector.go:560-565

Preview:

	for _, t := range tokens {
		_, _ = h.Write([]byte{byte(t.Kind)})
		if t.Text != "" {
			_, _ = h.Write([]byte(t.Text))
		}
		_, _ = h.Write([]byte{0}) // separator

View full run · Scanned by dupehound

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This PR introduces significant enhancements to dupehound's reporting and CLI functionality:

  1. Duplication Metrics: The Report structure now includes TotalLines and DuplicationPct (global duplication percentage). A new FileStats struct has been added to provide per-file duplication metrics, including TotalLines, DuplicateLines, and DuplicationPct for each file, sorted by duplication percentage.
  2. Quality Gate Flag: A new --min-duplication CLI flag (and corresponding ErrThresholdExceeded error) allows users to set a threshold for global duplication percentage. If the calculated duplication exceeds this value, the tool will exit with code 1, enabling its use as a quality gate in CI/CD pipelines.
  3. Output Enhancements: Text and Markdown reports have been updated to display the new global and per-file duplication statistics. The 'Hotspots' section now leverages FileStats and sorts files by duplication percentage. JSON output automatically includes all new fields.

Concerns

None. The changes are well-tested and logically sound.

Verdict

Approve


Code review performed by GEMINI - gemini-2.5-flash.

@machado144
machado144 merged commit cd0ac3d into main Apr 2, 2026
7 checks passed
@machado144
machado144 deleted the feat/per-file-duplication-stats branch April 2, 2026 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: per-file duplication stats and global duplication percentage

1 participant

⌂ 换网址