Repository navigation
feat: insightful output — function attribution, baseline ratchet, stable fingerprints, GitHub annotations - #27
Merged
Merged
Conversation
…thub format, insight metrics - Attribute every clone instance to its enclosing function/method across all 20 supported languages (keyword, brace-heuristic, def/end, arrow styles) - Replace position-based clone hashes with stable content-based fingerprints (full-block structural hash; order-independent pair hash for type-3) - Add --write-baseline / --baseline ratchet: record duplication debt, fail only on new clones or grown instance counts - Surface partial detection (--max-pairs / --max-bucket) in the report and all output formats instead of stderr-only warnings - Add per-clone refactor savings, scope (same-file/cross-file/cross-dir), and report-level refactor value - Add --format github: GitHub Actions inline annotations without SARIF setup - Add --max-file-size guard (default 5MiB) skipping oversized files before they are read into memory - 50 new tests: per-language name extraction, fingerprint stability regressions, ratchet semantics, partial surfacing, escaping, CLI flows
StructLint — All checks passed92 rules validated against
|
Manual audit of gin's 13 type-3 findings showed every clone pointed at real duplication, but ~40% were re-reports of the same region pair offset by a few lines: strict containment dedup (100% overlap against any kept range) misses one-line offsets. Replace it with pair-aware subsumption: a candidate is dropped only when ALL its instances are >=60% line-covered by instances of the SAME kept clone — i.e. it links exactly the regions an existing finding already links. The same-clone requirement is what makes the low threshold safe: a clone bridging two different kept clones' regions is new information and survives, as does any clone with an instance outside the kept set (the baseline ratchet's grown-instance signal). Results: gin 23->13 findings, prometheus 1958->1657, all remaining spot-checked findings real, determinism preserved (byte-identical reruns), performance unchanged. 5 new dedup regression tests.
dupehound scan results
Hotspots
Top clones (by impact)#1
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes dupehound's output actionable enough to live in the daily development lifecycle and hooks, and hardens the memory guards.
Function attribution (all 20 languages)
Every clone instance now names its enclosing function/method:
api/user.go:42-63 (in loadUser). Implemented per language family — keyword languages (Go incl. receivers/generics, Rust, Kotlin, Swift, Scala, PHP, shell), JS/TS (declarations, arrows incl. TS return-type annotations, class/object methods, function expressions), Python (def/async def), Ruby/Elixir/Lua (def…end, dotted/colon names), Java/C/C++/C#/Dart (brace heuristic with nested method spans), SQL, R (name <- function). Anonymous callbacks attribute to the innermost named enclosing function.Stable content-based fingerprints
Clone hashes are now derived from the normalized token structure of the full block (type-3: order-independent pair hash) instead of first-window / file-index+position. They survive line shifts, file renames, file additions, and unrelated edits — which finally makes hash-based suppression rules useful for type-3 and enables:
Baseline ratchet —
--write-baseline/--baselineAdopt dupehound on a legacy codebase without failing on day one:
--write-baseline .dupehound-baseline.jsonrecords current clones as accepted debt (exit 0), deterministic diff-friendly JSON--baseline …fails only on clones not in the baseline — or known clones whose instance count grew (pasting yet another copy still fails)N known, M NEW.dupehound.yml(scan.baseline)Partial results surfaced in the report
--max-pairs/--max-bucketcapping was stderr-only; JSON/md/SARIF consumers couldn't tell a capped scan from a complete one. Now:!! PARTIAL RESULTbanner (text), blockquote (md),partial/partial_reasonfields (json), run properties (SARIF), leading::warning(github). Type-1/2 completeness guarantee is stated explicitly.Insight metrics
saves ~N linesrefactor value + repo-wideRefactor valuesummary rowsame-file/cross-file/cross-dir(badges + JSON/SARIF fields)--format githubGitHub Actions workflow-command annotations (
::warning file=…,line=…,endLine=…::) with function attribution and duplicate locations in the message,::noticesummary, correct property/data escaping. Inline PR annotations from a single step, no SARIF upload.Memory guards (kept & extended)
--max-pairsdegradation and streaming pair evaluation kept intact (regression-tested for determinism)--max-file-size(default 5 MiB,0= unlimited): oversized files (minified/generated) are skipped before being read into memory, counted in the reportValidation on real OSS repos
On prometheus: baseline write is deterministic (byte-identical across runs, 52 KB for 1958 clones); ratchet compare exits 0 with
1958 known, 0 NEW; pasting one realistic duplicated function pair yields exactly1 NEW, exit 1, attributed to the right functions;--max-pairs 200000produces the partial banner with type-1/2 intact and peak RSS drops to 797 MB.Tests
+50 tests (358 total, race-clean, golangci-lint 0 issues):
async defafter another def — fixed a pre-existing Python marker bug this exposed)--max-file-sizeguard incl.0 = unlimitedPost-review quality audit (added)
Manually audited every gin type-3 finding and random prometheus type-3 samples against the actual source:
tree.gocase-matching blocks, prometheusAdd/Subshared prologues, repeated stale-marker handling, etc.), all function attributions correct.