Skip to content

feat: insightful output — function attribution, baseline ratchet, stable fingerprints, GitHub annotations - #27

Merged
machado144 merged 3 commits into
mainfrom
feat/insightful-output
Jul 3, 2026
Merged

machado144 merged 3 commits into
mainfrom
feat/insightful-output

Conversation

@machado144

@machado144 machado144 commented Jul 2, 2026 •

Copy link
Copy Markdown
Contributor

What

Makes dupehound's output actionable enough to live in the daily development lifecycle and hooks, and hardens the memory guards.

Function attribution (all 20 languages)

Every clone instance now names its enclosing function/method: api/user.go:42-63 (in loadUser). Implemented per language family — keyword languages (Go incl. receivers/generics, Rust, Kotlin, Swift, Scala, PHP, shell), JS/TS (declarations, arrows incl. TS return-type annotations, class/object methods, function expressions), Python (def/async def), Ruby/Elixir/Lua (def…end, dotted/colon names), Java/C/C++/C#/Dart (brace heuristic with nested method spans), SQL, R (name <- function). Anonymous callbacks attribute to the innermost named enclosing function.

Stable content-based fingerprints

Clone hashes are now derived from the normalized token structure of the full block (type-3: order-independent pair hash) instead of first-window / file-index+position. They survive line shifts, file renames, file additions, and unrelated edits — which finally makes hash-based suppression rules useful for type-3 and enables:

Baseline ratchet — --write-baseline / --baseline

Adopt dupehound on a legacy codebase without failing on day one:

  • --write-baseline .dupehound-baseline.json records current clones as accepted debt (exit 0), deterministic diff-friendly JSON
  • --baseline … fails only on clones not in the baseline — or known clones whose instance count grew (pasting yet another copy still fails)
  • known clones are hidden from output; summary shows N known, M NEW
  • configurable via .dupehound.yml (scan.baseline)

Partial results surfaced in the report

--max-pairs / --max-bucket capping was stderr-only; JSON/md/SARIF consumers couldn't tell a capped scan from a complete one. Now: !! PARTIAL RESULT banner (text), blockquote (md), partial/partial_reason fields (json), run properties (SARIF), leading ::warning (github). Type-1/2 completeness guarantee is stated explicitly.

Insight metrics

  • per-clone saves ~N lines refactor value + repo-wide Refactor value summary row
  • clone scope: same-file / cross-file / cross-dir (badges + JSON/SARIF fields)

--format github

GitHub Actions workflow-command annotations (::warning file=…,line=…,endLine=…::) with function attribution and duplicate locations in the message, ::notice summary, correct property/data escaping. Inline PR annotations from a single step, no SARIF upload.

Memory guards (kept & extended)

  • graceful --max-pairs degradation and streaming pair evaluation kept intact (regression-tested for determinism)
  • new --max-file-size (default 5 MiB, 0 = unlimited): oversized files (minified/generated) are skipped before being read into memory, counted in the report

Validation on real OSS repos

repo lines wall peak RSS result
gin 7.9k 0.23s 25 MB 23 clones, attribution correct
flask 18k 0.49s 40 MB Python attribution correct
express 21k 4.5s 104 MB clones in anonymous mocha callbacks → correctly unattributed
prometheus 194k 11.5s 959 MB 1958 clones; two runs byte-identical

On prometheus: baseline write is deterministic (byte-identical across runs, 52 KB for 1958 clones); ratchet compare exits 0 with 1958 known, 0 NEW; pasting one realistic duplicated function pair yields exactly 1 NEW, exit 1, attributed to the right functions; --max-pairs 200000 produces the partial banner with type-1/2 intact and peak RSS drops to 797 MB.

Tests

+50 tests (358 total, race-clean, golangci-lint 0 issues):

  • per-language name extraction incl. edge cases (Go anonymous func with return type, TS annotated arrows, async def after another def — fixed a pre-existing Python marker bug this exposed)
  • fingerprint stability regressions: line shift, file rename, unrelated-file addition (the old type-3 failure mode)
  • ratchet semantics: known/new, grown instance counts, suppressed exclusion, corrupt/missing/wrong-version baselines
  • partial surfacing through scanner and all formats; --max-file-size guard incl. 0 = unlimited
  • github format: command validity, property vs data escaping, top-cap overflow, dead functions, baseline summary
  • CLI end-to-end: full ratchet flow, line-shift survival, exit codes (0/1/2), function names and savings in text output
  • golden files extended with the new rendering

Post-review quality audit (added)

Manually audited every gin type-3 finding and random prometheus type-3 samples against the actual source:

  • 0 false positives — every finding points at genuinely duplicated logic (gin tree.go case-matching blocks, prometheus Add/Sub shared prologues, repeated stale-marker handling, etc.), all function attributions correct.
  • The audit exposed same-region re-reports (~40% of gin's type-3 list was the same region pair offset by 1–2 lines, defeating the strict-containment dedup). Fixed with pair-aware subsumption: a finding is dropped only when all its instances are ≥60% covered by the same kept clone. gin 23→13 findings, prometheus 1958→1657, with regression tests proving bridging clones and grown-instance clones still survive.
  • Determinism re-verified post-change (byte-identical reruns on prometheus).

…thub format, insight metrics

- Attribute every clone instance to its enclosing function/method across all
  20 supported languages (keyword, brace-heuristic, def/end, arrow styles)
- Replace position-based clone hashes with stable content-based fingerprints
  (full-block structural hash; order-independent pair hash for type-3)
- Add --write-baseline / --baseline ratchet: record duplication debt, fail
  only on new clones or grown instance counts
- Surface partial detection (--max-pairs / --max-bucket) in the report and
  all output formats instead of stderr-only warnings
- Add per-clone refactor savings, scope (same-file/cross-file/cross-dir),
  and report-level refactor value
- Add --format github: GitHub Actions inline annotations without SARIF setup
- Add --max-file-size guard (default 5MiB) skipping oversized files before
  they are read into memory
- 50 new tests: per-language name extraction, fingerprint stability
  regressions, ratchet semantics, partial surfacing, escaping, CLI flows
@github-actions

github-actions Bot commented Jul 2, 2026 •

Copy link
Copy Markdown

StructLint — All checks passed

92 rules validated against .structlint.yaml. No violations found.

View full run · Powered by StructLint

Manual audit of gin's 13 type-3 findings showed every clone pointed at real
duplication, but ~40% were re-reports of the same region pair offset by a few
lines: strict containment dedup (100% overlap against any kept range) misses
one-line offsets.

Replace it with pair-aware subsumption: a candidate is dropped only when ALL
its instances are >=60% line-covered by instances of the SAME kept clone —
i.e. it links exactly the regions an existing finding already links. The
same-clone requirement is what makes the low threshold safe: a clone bridging
two different kept clones' regions is new information and survives, as does
any clone with an instance outside the kept set (the baseline ratchet's
grown-instance signal).

Results: gin 23->13 findings, prometheus 1958->1657, all remaining spot-checked
findings real, determinism preserved (byte-identical reruns), performance
unchanged. 5 new dedup regression tests.
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown

dupehound scan results

Scanning diff since: origin/main (30 files changed)

Metric Value
Files scanned 19 / 19
Total lines 4531
Clones found 12
Duplicate lines 212 (4.7%)
Refactor value ~108 lines removable
New clones since origin/main 9
Breakdown 12 type-2

Hotspots

Dup % Dup Lines Total Lines File
19.4% 14 72 services/config.go
17.1% 148 863 services/reporter.go
6.1% 5 82 services/gitchurn.go
4.6% 10 218 services/funcname.go
2.9% 23 804 services/detector.go
1.4% 12 853 services/scanner.go

Top clones (by impact)

#1   type-2   similarity: 1.00   20 lines   2 instances   💾 ~20 lines   📁 same-file

Locations:

  • services/reporter.go:270-289 — in formatText
  • services/reporter.go:429-449 — in formatMarkdown

Preview:

			fmt.Fprintf(&b, "  ... %d more files (use --verbose to show all)\n", len(report.FileStats)-limit)
		}
	}

	if len(report.Clones) > 0 {
		testProdClones, normalClones := sortedPartition(report, opts)

		if len(testProdClones) > 0 {
			fmt.Fprintf(&b, "\nTest↔Prod clones (span test and production files):\n")
			writeClonesTextSection(&b, testProdClones, opts, "test↔prod clones")
// ... 10 more lines
#2   type-2   similarity: 1.00   14 lines   2 instances   💾 ~14 lines   📁 same-file

Locations:

  • services/reporter.go:248-261 — in formatText
  • services/reporter.go:394-408 — in formatMarkdown

Preview:

	for _, row := range summaryRows(report) {
		fmt.Fprintf(&b, "%s\n", row.text)
	}

	if report.TotalClones == 0 && len(report.DeadFunctions) == 0 {
		fmt.Fprintf(&b, "\nNo duplicates found.\n")
		return b.String()
	}

	if len(report.Clones) > 0 {
// ... 4 more lines
#3   type-2   similarity: 1.00   12 lines   2 instances   💾 ~12 lines   📁 same-file

Locations:

  • services/detector.go:189-200 — in detectExact
  • services/detector.go:581-591 — in detectFuzzy

Preview:

		if a.FileIdx != b.FileIdx {
			return a.FileIdx < b.FileIdx
		}
		return a.Pos < b.Pos
	})

	// Step 4: sweep left-to-right. For each uncovered candidate, gather its group,
	// extend the block as far as all members agree, then mark the block as covered.
	covered := make(map[globalPos]bool)
	var clones []domain.Clone
// ... 2 more lines
#4   type-2   similarity: 1.00   9 lines   2 instances   💾 ~9 lines   📁 same-file

Locations:

  • services/reporter.go:262-270 — in formatText
  • services/reporter.go:412-421 — in formatMarkdown

Preview:

		limit := topN(len(report.FileStats), defaultTopHotspots, opts.Verbose)
		fmt.Fprintf(&b, "\nHotspots (files by duplication %%):\n")
		for i := 0; i < limit; i++ {
			fs := report.FileStats[i]
			rel := relPath(fs.File, opts.ScanPath)
			fmt.Fprintf(&b, "  %5.1f%%  %4d/%4d lines  %s\n", fs.DuplicationPct, fs.DuplicateLines, fs.TotalLines, rel)
		}
		if len(report.FileStats) > limit {
			fmt.Fprintf(&b, "  ... %d more files (use --verbose to show all)\n", len(report.FileStats)-limit)
#5   type-2   similarity: 1.00   9 lines   2 instances   💾 ~9 lines   📁 same-file

Locations:

  • services/reporter.go:342-350 — in writeCloneTextWithTag
  • services/reporter.go:503-511 — in writeCloneMd

Preview:

		num, c.Type, c.Similarity, c.LineCount, c.TokenCount, len(c.Instances), savesLabel, testProdLabel, tag)
	instLimit := len(c.Instances)
	if !opts.Verbose && instLimit > maxInstancesShown {
		instLimit = maxInstancesShown
	}
	for i := 0; i < instLimit; i++ {
		inst := c.Instances[i]
		testLabel := ""
		if inst.IsTest {
#6   type-2   similarity: 1.00   8 lines   2 instances   💾 ~8 lines   📁 same-file

Locations:

  • services/reporter.go:165-172 — in summaryRows
  • services/reporter.go:180-187 — in summaryRows

Preview:

	if r.SkippedLargeFiles > 0 {
		rows = append(rows, metricRow{
			text: fmt.Sprintf("Skipped files : %d (over --max-file-size)", r.SkippedLargeFiles),
			md:   fmt.Sprintf("Skipped (oversized) | %d", r.SkippedLargeFiles),
		})
	}
	rows = append(rows, metricRow{
		text: fmt.Sprintf("Total lines   : %d", r.TotalLines),
#7   type-2   similarity: 1.00   7 lines   2 instances   💾 ~7 lines   📁 same-file

Locations:

  • services/reporter.go:241-247 — in formatText
  • services/reporter.go:385-391 — in formatMarkdown

Preview:

	fmt.Fprintf(&b, "======================\n")
	if report.Partial {
		fmt.Fprintf(&b, "!! PARTIAL RESULT: %s\n", report.PartialReason)
	}
	if report.SinceDiffRef != "" {
		fmt.Fprintf(&b, "Scanning diff  : since %s  (%d files changed)\n", report.SinceDiffRef, report.SinceDiffFiles)
	}
#8   type-2   similarity: 1.00   7 lines   2 instances   💾 ~7 lines   📁 same-file

Locations:

  • services/config.go:51-57 — in ApplyConfigDefaults
  • services/config.go:63-69 — in ApplyConfigDefaults

Preview:

	if opts.Path == "" && cfg.Scan.Path != "" {
		opts.Path = cfg.Scan.Path
	}
	if opts.MinTokens == 0 && cfg.Scan.MinTokens > 0 {
		opts.MinTokens = cfg.Scan.MinTokens
	}
	if len(opts.Exclude) == 0 && len(cfg.Scan.Exclude) > 0 {
#9   type-2   similarity: 1.00   6 lines   2 instances   💾 ~6 lines   📁 same-file

Locations:

  • services/scanner.go:715-720 — in deduplicateOverlapping
  • services/scanner.go:828-833 — in buildFileStats

Preview:

	sort.Slice(clones, func(i, j int) bool {
		if clones[i].LineCount != clones[j].LineCount {
			return clones[i].LineCount > clones[j].LineCount
		}
		return clones[i].Hash < clones[j].Hash
	})
#10   type-2   similarity: 1.00   6 lines   2 instances   💾 ~6 lines   📁 same-file

Locations:

  • services/reporter.go:369-374 — in writeCloneTextWithTag
  • services/reporter.go:534-539 — in writeCloneMd

Preview:

			for _, line := range lines[:limit] {
				fmt.Fprintf(b, "        | %s\n", strings.TrimRight(line, " \t"))
			}
			if len(lines) > previewMaxLines {
				fmt.Fprintf(b, "        ... (%d more lines)\n", len(lines)-previewMaxLines)
			}

2 more clones not shown. Run dupehound scan --verbose or --top 0 for full results.


View full run · Scanned by dupehound

@machado144
machado144 merged commit c92a850 into main Jul 3, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

⌂ 换网址