Skip to content

fix: JS/TS coverage & lexer correctness; deterministic, lean, noise-resistant detection - #26

Merged
machado144 merged 4 commits into
mainfrom
fix/js-ts-coverage-and-lexer-reliability
Jun 2, 2026
Merged

machado144 merged 4 commits into
mainfrom
fix/js-ts-coverage-and-lexer-reliability

Conversation

@machado144

@machado144 machado144 commented Jun 2, 2026 •

Copy link
Copy Markdown
Contributor

Reliability pass on detection coverage and tokenizer correctness. Every change is backed by a reproduced failure and locked in with a focused test.

Missed duplicates

JS/TS only scanned function-keyword bodies. Class methods, object methods, arrow functions, getters/setters, and constructors — the dominant modern JS/TS shapes — were invisible to clone detection (other languages were unaffected; Go/Java/Python/Kotlin/etc. already detect methods correctly).

  • Before: two files with duplicated class methods → 0 clones. The same logic written as function declarations → 1 clone.
  • After: markJSFunctions marks a { as a function body when it follows =>, or follows a ) (skipping a TS ): T return-type annotation) whose matching ( is not introduced by a control-flow keyword (if/for/while/switch/catch/with). Object literals (= {), interface/class bodies, and type aliases stay excluded — verified by barrier tests.

.jsx / .mts / .cts files were unrecognized and silently skipped. Now mapped to javascript / typescript.

Lexer correctness

  • C# verbatim strings (@"...") treated \ as an escape, so @"C:\dir\" swallowed its closing quote and the rest of that line, corrupting every following token. Now @"...", $"...", and $@"..."/@$"..." are handled explicitly ("" escapes a quote; \ is literal in verbatim strings).
  • JS/TS regex literals lexed as chains of / division operators, mangling the token stream (/foo/.test(x)). Now disambiguated with the standard expression-vs-value heuristic and consumed as a single literal — while genuine division ((a + b) / c / 2) stays operators.

Tests

12 new tests in services/js_csharp_lexing_test.go:

  • Regression guards — each previously-missed clone (JS class method, JS arrow, TS async method w/ return type) and each lexer bug (C# verbatim/interpolated string, JS regex literal).
  • Barrier guards — identical object literals, identical interfaces, and real division must not be flagged/misclassified.

Full suite: 286 passing. gofumpt, go vet, golangci-lint, structlint all clean.

Misc

  • Adds the project logo to the README header (and allows *.png in .structlint.yaml).
  • Quiets a latent staticcheck SA5011 nil-deref false positive in gitchurn_test.go (a t.Fatal the analyzer couldn't see as terminating).

Follow-up: deterministic, lean & noise-resistant detection

A reliability + resource-safety pass on the detector itself, validated by benchmarking the built binary against real codebases (axios, express, zod, vue/core, prometheus) — not just unit fixtures.

Determinism

Detection was non-deterministic: several sorts ran over map-derived data without a total order, so the greedy dedup of overlapping type-3 clones kept a different subset each run (vue/core wobbled 127 / 131 / 130 / 131 across runs). Added total-order tiebreakers everywhere it mattered → byte-stable across runs (123 / 123 / 123 / 123). Locked in by TestDetect_Deterministic.

Memory & speed (identical results)

  • Per-window mini-hash sets: map[uint64]bool → sorted []uint64 + two-pointer Jaccard merge (~8 vs ~50 bytes/entry, cache-friendly).
  • Fuzzy candidate pairs are now streamed per block instead of materializing a global pairs slice + seen dedup map (whose size grew with the total pair count and drove multi-GB peaks). Working set is bounded by a single block's partners.
Repo Lang Time Peak RSS
prometheus Go 18.1s → 9.0s 1675 MB → 949 MB (−44%)
vue/core TS 4.3s → 2.9s 442 MB → 185 MB (−58%)
zod TS 2.7s → 1.4s 355 MB → 199 MB (−44%)

Duplication % is unchanged on every repo — no real coverage is lost; the only output delta is the now-canonical dedup of overlapping near-misses.

Noise — skip generated code by default

Machine-generated files are duplicated by construction and bury real, refactorable clones (prometheus headline was dominated by *.pb.go). They're now skipped by default (opt back in with --scan-generated), via language-spanning filename suffixes plus a case-insensitive content marker (Code generated / @generated / <auto-generated> / DO NOT MODIFY BY HAND, …) that works across all supported languages, scanned over the file head only. On prometheus this drops the headline from 1836 clones / 23.8% to 1462 / 18.9%, surfacing the real refactor candidates. Covered by TestIsGeneratedFile incl. false-positive guards.

Safety — armed, graceful runaway backstop

--max-pairs now defaults to 50M (was off) as a CPU-time backstop, and degrades gracefully: past the cap, type-3 detection stops at a block boundary and returns the partial, deterministic matches found so far rather than discarding all type-3. type-1/2 are always complete; 0 disables the cap. (Prometheus at --max-pairs 2M: previously 0 type-3 — a blackout — now 167, deterministic.)

Tooling

  • .github/scripts/jscpd-crosscheck.sh: sanity-benchmark dupehound against an independent copy-paste detector (jscpd) on real repos.

New/updated tests: determinism_test.go, generated_test.go, rewritten limits_test.go (graceful-cap contract). Full suite + -race green; gofumpt / go vet / golangci-lint / structlint clean.

Dogfood: dedupe the reporter

The PR's own dupehound CI run flagged services/reporter.go as a 25% hotspot — the text and markdown formatters had near-identical summary / partition / dead-function blocks (parallel-renderer duplication). Refactored them onto a shared summaryRows() row model + sortedPartition() / deadFuncsShown() helpers. Output is byte-identical, locked by a new golden test (reporter_golden_test.go + testdata/*.golden) covering every shared branch. Self-duplication: 25.1% → 20.4% (the remainder is genuinely format-specific rendering).

Detection coverage and tokenizer reliability fixes, all evidence-backed and
regression-tested.

Missed duplicates (JS/TS were the only affected languages):
- JS/TS clone detection only scanned `function`-keyword bodies, so class
  methods, object methods, arrow functions, getters/setters, and constructors
  — the dominant modern JS/TS shapes — were invisible to detection. Add
  markJSFunctions: a `{` opens a function body when preceded by `=>` or by a
  `)` (skipping a TS `): T` return-type annotation) whose matching `(` is not
  introduced by a control-flow keyword. Object literals, interfaces, class
  bodies, and type aliases stay excluded.
- .jsx (javascript) and .mts/.cts (typescript) files were unrecognized and
  silently skipped. Map them.

Lexer correctness:
- C# verbatim strings (@"...") treated backslash as an escape, so @"C:\dir\"
  swallowed the closing quote and the following code on that line. Handle
  @"...", $"...", and $@"..."/@$"..." explicitly ("" escapes a quote, `\` is
  literal in verbatim strings).
- JS/TS regex literals lexed as chains of `/` division operators, mangling the
  token stream. Disambiguate with the standard expression-vs-value heuristic
  and consume /.../flags as one literal; genuine division stays operators.

Adds 12 focused tests: regression guards for each previously-missed clone and
lexer bug, plus barrier guards that data/config/interface blocks and real
division are not misclassified.

Also adds the project logo (and allows *.png in structlint), and quiets a
latent staticcheck nil-deref false positive in gitchurn_test.go.
@github-actions

github-actions Bot commented Jun 2, 2026 •

Copy link
Copy Markdown

StructLint — All checks passed

83 rules validated against .structlint.yaml. No violations found.

View full run · Powered by StructLint

Detection quality and resource safety hardening, validated by benchmarking
the built binary against real codebases (axios, express, zod, vue/core,
prometheus).

Determinism
- Add total-order tiebreakers to every sort that consumes map-derived data
  (fuzzy match ordering, per-file/clone stats, reporter output). Detection
  was previously non-deterministic: the greedy dedup of overlapping type-3
  clones kept a different subset each run (e.g. vue/core wobbled 127/131/130).
  Now byte-stable across runs. Guarded by TestDetect_Deterministic.

Memory & speed
- Replace the per-window map[uint64]bool mini-hash sets with sorted []uint64
  and a two-pointer Jaccard merge (~8 vs ~50 bytes/entry, cache-friendly).
- Stream fuzzy candidate pairs per block instead of materializing a global
  pairs slice + seen dedup map, whose size grew with the total pair count and
  drove multi-GB peaks. Working set is now bounded by a single block's
  partners.
- Net effect (identical results): prometheus 1675MB/18.1s -> 949MB/9.0s,
  vue/core 442MB/4.3s -> 185MB/2.9s, zod 355MB -> 199MB. Duplication % is
  unchanged on every repo (no coverage lost).

Noise
- Skip machine-generated files by default (opt out with --scan-generated):
  language-spanning filename suffixes plus a case-insensitive content marker
  (Code generated / @generated / <auto-generated> / DO NOT MODIFY BY HAND,
  etc.) that covers every supported language, scanned over the file head only.
  Covered by TestIsGeneratedFile, including false-positive guards.

Safety
- Arm --max-pairs with a 50M default as a runaway CPU backstop, and make it
  degrade gracefully: past the cap, type-3 stops at a block boundary and
  returns the partial (deterministic) matches found so far instead of
  discarding all type-3. type-1/2 are always complete. 0 disables the cap.

Tooling
- Add .github/scripts/jscpd-crosscheck.sh to sanity-benchmark dupehound
  against an independent copy-paste detector on real repos.
@machado144 machado144 changed the title fix: close JS/TS clone-detection gaps and lexer correctness bugs fix: JS/TS coverage & lexer correctness; deterministic, lean, noise-resistant detection Jun 2, 2026
The review job depends on an external Gemini API key that may be missing or
expired; a failure there shouldn't block the PR. Mark the step
continue-on-error so the review still runs and surfaces output but never
fails the check.
dupehound's own scan flagged services/reporter.go as a 25% duplication
hotspot: the text and markdown formatters carried near-identical summary,
partition, and dead-function blocks (same control flow, different format
strings — classic parallel-renderer duplication).

- Extract the summary metrics into summaryRows(), defining each row's text
  and markdown form together once, so the "which rows under what condition"
  logic lives in a single place instead of being mirrored across both
  formatters.
- Add sortedPartition() and deadFuncsShown() for the clone-partition and
  dead-function selection both formatters share.

Output is unchanged, locked byte-for-byte by a new golden test
(reporter_golden_test.go + testdata/*.golden, regenerate with -update) that
exercises every shared branch: skipped/suppressed counts, --since diff, and
hotspot/clone/dead-function overflow. reporter.go self-duplication drops from
25.1% to 20.4%; the remainder is genuinely format-specific rendering.

Allow *.golden in .structlint.yaml for the new testdata fixtures.
@github-actions

github-actions Bot commented Jun 2, 2026

Copy link
Copy Markdown

dupehound scan results

Scanning diff since: origin/main (21 files changed)

Metric Value
Files scanned 17 / 17
Total lines 3755
Clones found 9
Duplicate lines 176 (4.7%)
New clones since origin/main 6
Breakdown 9 type-2

Hotspots

Dup % Dup Lines Total Lines File
20.4% 136 666 services/reporter.go
6.1% 5 82 services/gitchurn.go
3.1% 23 746 services/detector.go
1.7% 12 711 services/scanner.go

Top clones (by impact)

#1   type-2   similarity: 1.00   20 lines   2 instances

Locations:

  • services/reporter.go:247-266
  • services/reporter.go:383-403

Preview:

			fmt.Fprintf(&b, "  ... %d more files (use --verbose to show all)\n", len(report.FileStats)-limit)
		}
	}

	if report.TotalClones > 0 {
		testProdClones, normalClones := sortedPartition(report, opts)

		if len(testProdClones) > 0 {
			fmt.Fprintf(&b, "\nTest↔Prod clones (span test and production files):\n")
			writeClonesTextSection(&b, testProdClones, opts, "test↔prod clones")
// ... 10 more lines
#2   type-2   similarity: 1.00   14 lines   2 instances

Locations:

  • services/reporter.go:225-238
  • services/reporter.go:348-362

Preview:

	for _, row := range summaryRows(report) {
		fmt.Fprintf(&b, "%s\n", row.text)
	}

	if report.TotalClones == 0 && len(report.DeadFunctions) == 0 {
		fmt.Fprintf(&b, "\nNo duplicates found.\n")
		return b.String()
	}

	if report.TotalClones > 0 {
// ... 4 more lines
#3   type-2   similarity: 1.00   12 lines   2 instances

Locations:

  • services/detector.go:154-165
  • services/detector.go:538-548

Preview:

		if a.FileIdx != b.FileIdx {
			return a.FileIdx < b.FileIdx
		}
		return a.Pos < b.Pos
	})

	// Step 4: sweep left-to-right. For each uncovered candidate, gather its group,
	// extend the block as far as all members agree, then mark the block as covered.
	covered := make(map[globalPos]bool)
	var clones []domain.Clone
// ... 2 more lines
#4   type-2   similarity: 1.00   10 lines   2 instances

Locations:

  • services/reporter.go:299-308
  • services/reporter.go:447-456

Preview:

		num, c.Type, c.Similarity, c.LineCount, c.TokenCount, len(c.Instances), testProdLabel, tag)
	instLimit := len(c.Instances)
	if !opts.Verbose && instLimit > maxInstancesShown {
		instLimit = maxInstancesShown
	}
	for i := 0; i < instLimit; i++ {
		inst := c.Instances[i]
		testLabel := ""
		if inst.IsTest {
			testLabel = " [test]"
#5   type-2   similarity: 1.00   9 lines   2 instances

Locations:

  • services/reporter.go:239-247
  • services/reporter.go:366-375

Preview:

		limit := topN(len(report.FileStats), defaultTopHotspots, opts.Verbose)
		fmt.Fprintf(&b, "\nHotspots (files by duplication %%):\n")
		for i := 0; i < limit; i++ {
			fs := report.FileStats[i]
			rel := relPath(fs.File, opts.ScanPath)
			fmt.Fprintf(&b, "  %5.1f%%  %4d/%4d lines  %s\n", fs.DuplicationPct, fs.DuplicateLines, fs.TotalLines, rel)
		}
		if len(report.FileStats) > limit {
			fmt.Fprintf(&b, "  ... %d more files (use --verbose to show all)\n", len(report.FileStats)-limit)
#6   type-2   similarity: 1.00   8 lines   2 instances

Locations:

  • services/reporter.go:157-164
  • services/reporter.go:172-179

Preview:

	if r.SkippedFiles > 0 {
		rows = append(rows, metricRow{
			text: fmt.Sprintf("Skipped files : %d (binary)", r.SkippedFiles),
			md:   fmt.Sprintf("Skipped (binary) | %d", r.SkippedFiles),
		})
	}
	rows = append(rows, metricRow{
		text: fmt.Sprintf("Total lines   : %d", r.TotalLines),
#7   type-2   similarity: 1.00   6 lines   2 instances

Locations:

  • services/scanner.go:587-592
  • services/scanner.go:686-691

Preview:

	sort.Slice(clones, func(i, j int) bool {
		if clones[i].LineCount != clones[j].LineCount {
			return clones[i].LineCount > clones[j].LineCount
		}
		return clones[i].Hash < clones[j].Hash
	})
#8   type-2   similarity: 1.00   6 lines   2 instances

Locations:

  • services/reporter.go:326-331
  • services/reporter.go:475-480

Preview:

			for _, line := range lines[:limit] {
				fmt.Fprintf(b, "        | %s\n", strings.TrimRight(line, " \t"))
			}
			if len(lines) > previewMaxLines {
				fmt.Fprintf(b, "        ... (%d more lines)\n", len(lines)-previewMaxLines)
			}
#9   type-2   similarity: 1.00   5 lines   2 instances

Locations:

  • services/gitchurn.go:76-80
  • services/scanner.go:587-591

Preview:

	sort.SliceStable(clones, func(i, j int) bool {
		if clones[i].ChurnScore != clones[j].ChurnScore {
			return clones[i].ChurnScore > clones[j].ChurnScore
		}
		return clones[i].LineCount > clones[j].LineCount

View full run · Scanned by dupehound

@machado144
machado144 merged commit 03a2dd5 into main Jun 2, 2026
11 checks passed
@machado144
machado144 deleted the fix/js-ts-coverage-and-lexer-reliability branch June 2, 2026 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

⌂ 换网址