Skip to content

feat!: whole-file clone detection, memory budget, validated dead code - #33

Merged
machado144 merged 20 commits into
mainfrom
fix/memory-blowup-and-lexer-accuracy
Oct 13, 2026
Merged

machado144 merged 20 commits into
mainfrom
fix/memory-blowup-and-lexer-accuracy

Conversation

@machado144

@machado144 machado144 commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Measured on real repositories against SonarQube, staticcheck, rustc, vulture and jscpd. Every fix started as a failing test.

Results

Clones. Share of Sonar's duplicated lines dupehound also reports, at default settings:

tokio nest guava terraform django laravel rails
99% 99% 99% 96% 93% 90% 89%

Before, forced to Sonar's 100-token threshold, it found 38% on tokio, 57% on laravel and 27% on rails.

Memory. terraform 7.5 GB → 0.7 GB. rails was OOM-killed at 7.8 GB, now 1.5 GB. kubernetes (2.95M lines) was killed at 4 GB; it now completes in 13 s, or under a 2 GB budget in 15 s with identical findings.

Dead code. Wrong findings: golang/tools vs staticcheck 36 of 37 → 1 of 4; tokio vs rustc 6 of 6 → 0. Findings on one real repo per language: Kotlin 2,273 → 21, Swift 2,416 → 74, Lua 476 → 1, redis (C) high-confidence 538 → 2.

All 20 languages have now been run on a real repository. No clone range runs past the end of its file in any of them.

This repository. dupehound and jscpd both report zero duplicates in its production code (was 20 and 5).

What changed

  • --scope file (new default): a clone may span functions or sit outside any. Outside function bodies names must match; imports are skipped.
  • --min-clone-lines 5 (new default): drops clones of a few dense lines.
  • --max-memory (default: half of system or container memory): a scan that would exceed it switches to a low-memory mode with identical type-1/2 results; type-3 is skipped and the report says so. If even that does not fit, it stops with a clear error.
  • Dead code: usages searched by visibility; framework-invoked functions skipped; methods found in Java, C, C++, C#, Dart and JS/TS classes; rules for Kotlin, Swift, Scala, Lua and Elixir; each finding is high or low confidence.
  • Lexer: line numbers no longer drift after unclosed quotes; heredocs, triple-quoted strings and Rust lifetimes are lexed correctly; Lua and Elixir blocks are delimited correctly.
  • Generated files: prose such as "generated by" in a doc comment no longer hides a hand-written file.
  • Code layout: files are <feature>.<concern>.go, enforced by structlint; CLAUDE.md maps each file to what it owns.

Breaking

  • Scans report more clones (file scope) and fewer tiny ones (line floor). Baselines will show the difference as new. --scope function --min-clone-lines 0 restores the old behaviour.
  • --dead-code exits 1 only for high confidence findings.
  • With little memory available, type-3 detection is skipped; the report says so. --max-memory 0 disables the budget.

Known limits

  • The duplication percentage is not comparable to Sonar's: at 50 tokens with renamed copies counted, dupehound reports 14–44% where Sonar reports 1–20%.
  • Low-memory mode is about a third of the memory, not disk spill: kubernetes at a 1 GB budget still stops with an error.
  • Default-mode (type-3) memory is unchanged.
  • Dead code remains name matching. It cannot see //go:linkname, names built at runtime, or macro token pasting.

Full numbers: docs/specs/002-scale-and-recall.md.

Testing

go test -race ./..., golangci-lint, govulncheck and the repo's own gate pass. Refactors were verified by byte-identical output on real repositories; new rules by removing each and confirming its test fails.

* Preview lines were substrings of the file just read, so every clone
  instance kept a private copy of its entire file alive
* Peak RSS on hashicorp/terraform: 7.5 GB -> 1.1 GB, identical findings
* A quote left open on its line counted that newline twice, so every
  later clone in the file was reported on the wrong lines
* Lex Rust lifetimes as punctuation instead of char literals
* Lex triple-quoted strings in Kotlin, Scala, Swift, Java, C#, Dart, Elixir
* Count backslash-newline continuations inside strings
* Recognise POSIX shell functions written as name() { ... }
* Skip main/init, test and benchmark functions, xUnit hooks, exported Go
  methods, decorated Python functions and Rust #[test] functions
* Test functions were 96% of findings on terraform and 99% on django
* Prose such as "the hash code is generated by" in a doc comment made
  hand-written files vanish from the scan (20 files in google/guava)
* Unambiguous markers (@generated, DO NOT EDIT) still match anywhere in
  the first 4 KB
* Records the seven-repository evaluation against SonarQube
* Proposes a memory budget with disk spill and detection beyond
  single function bodies; neither is implemented yet
* Search usages by visibility: unexported Go in its package, private
  Rust in its crate tree, module-local JS/TS in its file
* Count a call on the definition's own line, calls from skipped or
  generated files, names in strings, templates and config
* Skip Rust trait methods, Python overrides of external base classes,
  function literals; find JS arrow and function-expression definitions
* Stream usage files one at a time instead of holding the whole tree
* Label findings high or low confidence; analyse files below min-tokens
* Skip minified bundles; match loose generated phrases per header line

Scored on golang/tools vs staticcheck: 36 of 37 wrong -> 1 of 4.
tokio vs rustc: 6 of 6 wrong -> 0 findings.
* --max-memory (default: half of system or container memory) skips
  type-3 detection when its index would not fit and stops with a clear
  error when the scan cannot fit at all, instead of being OOM-killed
* Rolling window hash and an index of repeated hashes only: exact mode
  is 20-30% faster and 15-27% smaller, output identical on seven repos
* Hash each fuzzy mini-window once per file instead of once per block
* kubernetes (2.9M lines) now completes in exact mode under 4 GB
* Unused public functions are still reported but no longer exit 1
* Walk the tree for usages with filepath.Walk, as collectFiles does
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

✅ PR title follows the required format

Current title: feat!: whole-file clone detection, memory budget, validated dead code

* --scope file is the new default; --scope function keeps the old behaviour
* Outside function bodies names must match and imports are skipped, so
  declarations and data tables of the same shape are not reported
* Lex Ruby, shell and PHP heredocs as one string literal
* Attribute a clone to the function it overlaps most

Share of Sonar's duplicated lines found, function -> file scope:
tokio 38 -> 96%, laravel 57 -> 89%, django 77 -> 92%, rails 27 -> 76%.
@machado144 machado144 changed the title feat: memory budget, validated dead-code detection and line-number fixes feat!: whole-file clone detection, memory budget and validated dead-code detection Oct 13, 2026
* tokenizer.go (1,994 lines) into lexer, keywords, strings, function
  bodies, data literals, imports and inline-ignore files
* detector.go into entry point, tokenized file, exact, fuzzy, instances
* scanner.go into pipeline, file collection, generated detection, results
* deadcode.go into analysis and definition extraction
* Declarations moved verbatim; scan output is byte-identical
* One nested-body marker for brace, BEGIN/END and def/end blocks
* One clone filter, shared bracket-group and namespace helpers
* Sort comparators written with cmp.Or
* Scan output is byte-identical across json, text, md and sarif
…i-line strings

* --min-clone-lines (default 5) drops clones of a few dense lines, which
  were 0.1-8% of findings on seven repositories
* A clone ending in a heredoc or raw string now covers that string's lines
* Definitions are recognised as name(params) { outside executable code,
  replacing the old 'identifier before a paren' guess
* Visibility from modifiers: Java private, C/C++ static, JS/TS private
* Skip annotated, overriding and virtual members, serialization hooks,
  and methods of derived JS/TS classes
* C statics in headers, in included sources and in files that paste
  tokens are not treated as file-local

redis: 538 high-confidence findings -> 2; guava 16, abseil 21, nest 4,
none with another mention in scope.
…y format

* Every Go file follows <feature>.<concern>.go; structlint enforces it
* reporter.go becomes one file per output format on shared helpers that
  decide what a report shows
* Remaining duplication removed: dupehound and jscpd both report none
  in this repository's production code (was 20 and 5 clones)
* Output is byte-identical across text, markdown, json and sarif
@machado144 machado144 changed the title feat!: whole-file clone detection, memory budget and validated dead-code detection feat!: whole-file clone detection, memory budget, validated dead code Oct 13, 2026
…guages

* Lua, Elixir: count only the keywords an end actually closes, so a
  function body no longer runs to the end of the file
* Kotlin, Scala, Swift: skip annotated, overriding, implicit and XCTest
  functions; private ones are file-scoped; Kotlin extension receivers
* JavaScript: named function expressions are not callable definitions
* A function with an empty body no longer borrows the next block

One real repository per language: Kotlin 2,273 -> 21 findings,
Swift 2,416 -> 74, Scala 1,094 -> 114, Lua 476 -> 1, Elixir 60 -> 19.
* When holding every token would exceed --max-memory, keep each file as
  a compact index (window hash and line per token) and re-read a file
  only where a clone needs its text
* Type-1/2 findings and dead functions are identical to an unconstrained
  scan on seven repositories and on kubernetes; type-3 is skipped
* kubernetes (2.95M lines) under a 2 GB budget: 14.6 s, 1.76 GB peak,
  where it previously stopped with an error
@machado144
machado144 merged commit e43ea75 into main Oct 13, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

⌂ 换网址