Repository navigation
feat!: whole-file clone detection, memory budget, validated dead code - #33
Merged
Merged
Conversation
* Preview lines were substrings of the file just read, so every clone instance kept a private copy of its entire file alive * Peak RSS on hashicorp/terraform: 7.5 GB -> 1.1 GB, identical findings
* A quote left open on its line counted that newline twice, so every
later clone in the file was reported on the wrong lines
* Lex Rust lifetimes as punctuation instead of char literals
* Lex triple-quoted strings in Kotlin, Scala, Swift, Java, C#, Dart, Elixir
* Count backslash-newline continuations inside strings
* Recognise POSIX shell functions written as name() { ... }
* Skip main/init, test and benchmark functions, xUnit hooks, exported Go methods, decorated Python functions and Rust #[test] functions * Test functions were 96% of findings on terraform and 99% on django
* Prose such as "the hash code is generated by" in a doc comment made hand-written files vanish from the scan (20 files in google/guava) * Unambiguous markers (@generated, DO NOT EDIT) still match anywhere in the first 4 KB
* Records the seven-repository evaluation against SonarQube * Proposes a memory budget with disk spill and detection beyond single function bodies; neither is implemented yet
* Search usages by visibility: unexported Go in its package, private Rust in its crate tree, module-local JS/TS in its file * Count a call on the definition's own line, calls from skipped or generated files, names in strings, templates and config * Skip Rust trait methods, Python overrides of external base classes, function literals; find JS arrow and function-expression definitions * Stream usage files one at a time instead of holding the whole tree * Label findings high or low confidence; analyse files below min-tokens * Skip minified bundles; match loose generated phrases per header line Scored on golang/tools vs staticcheck: 36 of 37 wrong -> 1 of 4. tokio vs rustc: 6 of 6 wrong -> 0 findings.
* --max-memory (default: half of system or container memory) skips type-3 detection when its index would not fit and stops with a clear error when the scan cannot fit at all, instead of being OOM-killed * Rolling window hash and an index of repeated hashes only: exact mode is 20-30% faster and 15-27% smaller, output identical on seven repos * Hash each fuzzy mini-window once per file instead of once per block * kubernetes (2.9M lines) now completes in exact mode under 4 GB
* Unused public functions are still reported but no longer exit 1 * Walk the tree for usages with filepath.Walk, as collectFiles does
✅ PR title follows the required formatCurrent title: |
* --scope file is the new default; --scope function keeps the old behaviour * Outside function bodies names must match and imports are skipped, so declarations and data tables of the same shape are not reported * Lex Ruby, shell and PHP heredocs as one string literal * Attribute a clone to the function it overlaps most Share of Sonar's duplicated lines found, function -> file scope: tokio 38 -> 96%, laravel 57 -> 89%, django 77 -> 92%, rails 27 -> 76%.
* tokenizer.go (1,994 lines) into lexer, keywords, strings, function bodies, data literals, imports and inline-ignore files * detector.go into entry point, tokenized file, exact, fuzzy, instances * scanner.go into pipeline, file collection, generated detection, results * deadcode.go into analysis and definition extraction * Declarations moved verbatim; scan output is byte-identical
* One nested-body marker for brace, BEGIN/END and def/end blocks * One clone filter, shared bracket-group and namespace helpers * Sort comparators written with cmp.Or * Scan output is byte-identical across json, text, md and sarif
…i-line strings * --min-clone-lines (default 5) drops clones of a few dense lines, which were 0.1-8% of findings on seven repositories * A clone ending in a heredoc or raw string now covers that string's lines
* Definitions are recognised as name(params) { outside executable code,
replacing the old 'identifier before a paren' guess
* Visibility from modifiers: Java private, C/C++ static, JS/TS private
* Skip annotated, overriding and virtual members, serialization hooks,
and methods of derived JS/TS classes
* C statics in headers, in included sources and in files that paste
tokens are not treated as file-local
redis: 538 high-confidence findings -> 2; guava 16, abseil 21, nest 4,
none with another mention in scope.
…y format * Every Go file follows <feature>.<concern>.go; structlint enforces it * reporter.go becomes one file per output format on shared helpers that decide what a report shows * Remaining duplication removed: dupehound and jscpd both report none in this repository's production code (was 20 and 5 clones) * Output is byte-identical across text, markdown, json and sarif
…guages * Lua, Elixir: count only the keywords an end actually closes, so a function body no longer runs to the end of the file * Kotlin, Scala, Swift: skip annotated, overriding, implicit and XCTest functions; private ones are file-scoped; Kotlin extension receivers * JavaScript: named function expressions are not callable definitions * A function with an empty body no longer borrows the next block One real repository per language: Kotlin 2,273 -> 21 findings, Swift 2,416 -> 74, Scala 1,094 -> 114, Lua 476 -> 1, Elixir 60 -> 19.
* When holding every token would exceed --max-memory, keep each file as a compact index (window hash and line per token) and re-read a file only where a clone needs its text * Type-1/2 findings and dead functions are identical to an unconstrained scan on seven repositories and on kubernetes; type-3 is skipped * kubernetes (2.95M lines) under a 2 GB budget: 14.6 s, 1.76 GB peak, where it previously stopped with an error
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measured on real repositories against SonarQube,
staticcheck,rustc,vultureand jscpd. Every fix started as a failing test.Results
Clones. Share of Sonar's duplicated lines dupehound also reports, at default settings:
Before, forced to Sonar's 100-token threshold, it found 38% on tokio, 57% on laravel and 27% on rails.
Memory. terraform 7.5 GB → 0.7 GB. rails was OOM-killed at 7.8 GB, now 1.5 GB. kubernetes (2.95M lines) was killed at 4 GB; it now completes in 13 s, or under a 2 GB budget in 15 s with identical findings.
Dead code. Wrong findings: golang/tools vs
staticcheck36 of 37 → 1 of 4; tokio vsrustc6 of 6 → 0. Findings on one real repo per language: Kotlin 2,273 → 21, Swift 2,416 → 74, Lua 476 → 1, redis (C) high-confidence 538 → 2.All 20 languages have now been run on a real repository. No clone range runs past the end of its file in any of them.
This repository. dupehound and jscpd both report zero duplicates in its production code (was 20 and 5).
What changed
--scope file(new default): a clone may span functions or sit outside any. Outside function bodies names must match; imports are skipped.--min-clone-lines 5(new default): drops clones of a few dense lines.--max-memory(default: half of system or container memory): a scan that would exceed it switches to a low-memory mode with identical type-1/2 results; type-3 is skipped and the report says so. If even that does not fit, it stops with a clear error.highorlowconfidence.<feature>.<concern>.go, enforced by structlint;CLAUDE.mdmaps each file to what it owns.Breaking
--scope function --min-clone-lines 0restores the old behaviour.--dead-codeexits 1 only forhighconfidence findings.--max-memory 0disables the budget.Known limits
//go:linkname, names built at runtime, or macro token pasting.Full numbers:
docs/specs/002-scale-and-recall.md.Testing
go test -race ./...,golangci-lint,govulncheckand the repo's own gate pass. Refactors were verified by byte-identical output on real repositories; new rules by removing each and confirming its test fails.