Repository navigation
fix: JS/TS coverage & lexer correctness; deterministic, lean, noise-resistant detection - #26
Merged
Merged
Conversation
Detection coverage and tokenizer reliability fixes, all evidence-backed and
regression-tested.
Missed duplicates (JS/TS were the only affected languages):
- JS/TS clone detection only scanned `function`-keyword bodies, so class
methods, object methods, arrow functions, getters/setters, and constructors
— the dominant modern JS/TS shapes — were invisible to detection. Add
markJSFunctions: a `{` opens a function body when preceded by `=>` or by a
`)` (skipping a TS `): T` return-type annotation) whose matching `(` is not
introduced by a control-flow keyword. Object literals, interfaces, class
bodies, and type aliases stay excluded.
- .jsx (javascript) and .mts/.cts (typescript) files were unrecognized and
silently skipped. Map them.
Lexer correctness:
- C# verbatim strings (@"...") treated backslash as an escape, so @"C:\dir\"
swallowed the closing quote and the following code on that line. Handle
@"...", $"...", and $@"..."/@$"..." explicitly ("" escapes a quote, `\` is
literal in verbatim strings).
- JS/TS regex literals lexed as chains of `/` division operators, mangling the
token stream. Disambiguate with the standard expression-vs-value heuristic
and consume /.../flags as one literal; genuine division stays operators.
Adds 12 focused tests: regression guards for each previously-missed clone and
lexer bug, plus barrier guards that data/config/interface blocks and real
division are not misclassified.
Also adds the project logo (and allows *.png in structlint), and quiets a
latent staticcheck nil-deref false positive in gitchurn_test.go.
StructLint — All checks passed83 rules validated against
|
Detection quality and resource safety hardening, validated by benchmarking the built binary against real codebases (axios, express, zod, vue/core, prometheus). Determinism - Add total-order tiebreakers to every sort that consumes map-derived data (fuzzy match ordering, per-file/clone stats, reporter output). Detection was previously non-deterministic: the greedy dedup of overlapping type-3 clones kept a different subset each run (e.g. vue/core wobbled 127/131/130). Now byte-stable across runs. Guarded by TestDetect_Deterministic. Memory & speed - Replace the per-window map[uint64]bool mini-hash sets with sorted []uint64 and a two-pointer Jaccard merge (~8 vs ~50 bytes/entry, cache-friendly). - Stream fuzzy candidate pairs per block instead of materializing a global pairs slice + seen dedup map, whose size grew with the total pair count and drove multi-GB peaks. Working set is now bounded by a single block's partners. - Net effect (identical results): prometheus 1675MB/18.1s -> 949MB/9.0s, vue/core 442MB/4.3s -> 185MB/2.9s, zod 355MB -> 199MB. Duplication % is unchanged on every repo (no coverage lost). Noise - Skip machine-generated files by default (opt out with --scan-generated): language-spanning filename suffixes plus a case-insensitive content marker (Code generated / @generated / <auto-generated> / DO NOT MODIFY BY HAND, etc.) that covers every supported language, scanned over the file head only. Covered by TestIsGeneratedFile, including false-positive guards. Safety - Arm --max-pairs with a 50M default as a runaway CPU backstop, and make it degrade gracefully: past the cap, type-3 stops at a block boundary and returns the partial (deterministic) matches found so far instead of discarding all type-3. type-1/2 are always complete. 0 disables the cap. Tooling - Add .github/scripts/jscpd-crosscheck.sh to sanity-benchmark dupehound against an independent copy-paste detector on real repos.
The review job depends on an external Gemini API key that may be missing or expired; a failure there shouldn't block the PR. Mark the step continue-on-error so the review still runs and surfaces output but never fails the check.
dupehound's own scan flagged services/reporter.go as a 25% duplication hotspot: the text and markdown formatters carried near-identical summary, partition, and dead-function blocks (same control flow, different format strings — classic parallel-renderer duplication). - Extract the summary metrics into summaryRows(), defining each row's text and markdown form together once, so the "which rows under what condition" logic lives in a single place instead of being mirrored across both formatters. - Add sortedPartition() and deadFuncsShown() for the clone-partition and dead-function selection both formatters share. Output is unchanged, locked byte-for-byte by a new golden test (reporter_golden_test.go + testdata/*.golden, regenerate with -update) that exercises every shared branch: skipped/suppressed counts, --since diff, and hotspot/clone/dead-function overflow. reporter.go self-duplication drops from 25.1% to 20.4%; the remainder is genuinely format-specific rendering. Allow *.golden in .structlint.yaml for the new testdata fixtures.
dupehound scan results
Hotspots
Top clones (by impact)#1
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reliability pass on detection coverage and tokenizer correctness. Every change is backed by a reproduced failure and locked in with a focused test.
Missed duplicates
JS/TS only scanned
function-keyword bodies. Class methods, object methods, arrow functions, getters/setters, and constructors — the dominant modern JS/TS shapes — were invisible to clone detection (other languages were unaffected; Go/Java/Python/Kotlin/etc. already detect methods correctly).functiondeclarations → 1 clone.markJSFunctionsmarks a{as a function body when it follows=>, or follows a)(skipping a TS): Treturn-type annotation) whose matching(is not introduced by a control-flow keyword (if/for/while/switch/catch/with). Object literals (= {),interface/classbodies, and type aliases stay excluded — verified by barrier tests..jsx/.mts/.ctsfiles were unrecognized and silently skipped. Now mapped to javascript / typescript.Lexer correctness
@"...") treated\as an escape, so@"C:\dir\"swallowed its closing quote and the rest of that line, corrupting every following token. Now@"...",$"...", and$@"..."/@$"..."are handled explicitly (""escapes a quote;\is literal in verbatim strings)./division operators, mangling the token stream (/foo/.test(x)). Now disambiguated with the standard expression-vs-value heuristic and consumed as a single literal — while genuine division ((a + b) / c / 2) stays operators.Tests
12 new tests in
services/js_csharp_lexing_test.go:Full suite: 286 passing.
gofumpt,go vet,golangci-lint,structlintall clean.Misc
*.pngin.structlint.yaml).staticcheckSA5011 nil-deref false positive ingitchurn_test.go(at.Fatalthe analyzer couldn't see as terminating).Follow-up: deterministic, lean & noise-resistant detection
A reliability + resource-safety pass on the detector itself, validated by benchmarking the built binary against real codebases (axios, express, zod, vue/core, prometheus) — not just unit fixtures.
Determinism
Detection was non-deterministic: several sorts ran over map-derived data without a total order, so the greedy dedup of overlapping type-3 clones kept a different subset each run (vue/core wobbled
127 / 131 / 130 / 131across runs). Added total-order tiebreakers everywhere it mattered → byte-stable across runs (123 / 123 / 123 / 123). Locked in byTestDetect_Deterministic.Memory & speed (identical results)
map[uint64]bool→ sorted[]uint64+ two-pointer Jaccard merge (~8 vs ~50 bytes/entry, cache-friendly).pairsslice +seendedup map (whose size grew with the total pair count and drove multi-GB peaks). Working set is bounded by a single block's partners.Duplication % is unchanged on every repo — no real coverage is lost; the only output delta is the now-canonical dedup of overlapping near-misses.
Noise — skip generated code by default
Machine-generated files are duplicated by construction and bury real, refactorable clones (prometheus headline was dominated by
*.pb.go). They're now skipped by default (opt back in with--scan-generated), via language-spanning filename suffixes plus a case-insensitive content marker (Code generated/@generated/<auto-generated>/DO NOT MODIFY BY HAND, …) that works across all supported languages, scanned over the file head only. On prometheus this drops the headline from1836 clones / 23.8%to1462 / 18.9%, surfacing the real refactor candidates. Covered byTestIsGeneratedFileincl. false-positive guards.Safety — armed, graceful runaway backstop
--max-pairsnow defaults to 50M (was off) as a CPU-time backstop, and degrades gracefully: past the cap, type-3 detection stops at a block boundary and returns the partial, deterministic matches found so far rather than discarding all type-3. type-1/2 are always complete;0disables the cap. (Prometheus at--max-pairs 2M: previously0type-3 — a blackout — now167, deterministic.)Tooling
.github/scripts/jscpd-crosscheck.sh: sanity-benchmark dupehound against an independent copy-paste detector (jscpd) on real repos.New/updated tests:
determinism_test.go,generated_test.go, rewrittenlimits_test.go(graceful-cap contract). Full suite +-racegreen;gofumpt/go vet/golangci-lint/structlintclean.Dogfood: dedupe the reporter
The PR's own dupehound CI run flagged
services/reporter.goas a 25% hotspot — the text and markdown formatters had near-identical summary / partition / dead-function blocks (parallel-renderer duplication). Refactored them onto a sharedsummaryRows()row model +sortedPartition()/deadFuncsShown()helpers. Output is byte-identical, locked by a new golden test (reporter_golden_test.go+testdata/*.golden) covering every shared branch. Self-duplication: 25.1% → 20.4% (the remainder is genuinely format-specific rendering).