Repository navigation
feat(#3): clone type labels and similarity scores - #9
Conversation
Classify each detected clone as type-1 (token-for-token identical) or type-2 (same structure, differing identifiers/literals) with a similarity score. Adds OrigText to Token for post-detection comparison, updates text, JSON, and SARIF formatters, and includes 10 new tests (5 unit + 5 integration).
StructLint — All checks passed46 rules validated against
|
There was a problem hiding this comment.
Core Changes
This Pull Request introduces the classification of detected code clones into 'type-1' and 'type-2', along with a similarity score.
Key changes include:
- Data Model Enhancement: The
domain.Clonestruct now includesType(string) andSimilarity(float64) fields, with constants defined forCloneType1,CloneType2, andCloneType3. - Lexer Update: The
services/tokenizer.gonow stores the original source text (OrigText) forTokIdent,TokNumber, andTokStringtokens. ThisOrigTextis crucial for post-detection classification without affecting the structural hashing. - Clone Classification Logic: A new
classifyClonefunction inservices/detector.gocompares theOrigTextof tokens across clone instances to determine if they are type-1 (token-for-token identical) or type-2 (structurally identical but with differing identifiers/literals). For both type-1 and type-2, the similarity is currently set to 1.0. - Output Format Updates: All output formats (text, JSON, SARIF) have been updated to display the new clone type and similarity score.
- Comprehensive Testing: New unit tests in
services/detector_test.goand integration tests inintegration/cli_test.govalidate the correctness of clone type classification and its representation in various outputs.
Concerns (if any)
None. The changes are well-isolated, thoroughly tested, and align with the project's goals for clone detection.
Verdict
Approve
Code review performed by GEMINI - gemini-2.5-flash.
| Lines []string `json:"lines"` | ||
| } | ||
|
|
||
| // Clone type constants. |
There was a problem hiding this comment.
[SUGGESTION] The CloneType3 constant is defined but the PR summary mentions it's reserved for future work (#4). It's good to have it defined early, but ensure that the current implementation doesn't accidentally report type-3 clones or misrepresent their similarity if the fuzzy detector isn't yet integrated. The current classifyClone function correctly defaults to type-1 or type-2.
| @@ -222,6 +226,31 @@ func Detect(files []TokenizedFile, minTokens int) []domain.Clone { | |||
| return clones | |||
| } | |||
There was a problem hiding this comment.
[SUGGESTION] The classifyClone function includes a check if len(starts) < 2 { return domain.CloneType1, 1.0 }. This condition is redundant because the Detect function already ensures that len(starts) is at least 2 before calling classifyClone (lines 179-181). While harmless, it could be removed for cleaner code.
| // `if` ≠ `for` and `+` ≠ `-` in clone comparisons. | ||
| // OrigText preserves the original source text for Ident/Number/String tokens, | ||
| // used after detection to classify clones as type-1 vs type-2. | ||
| type Token struct { |
There was a problem hiding this comment.
[SUGGESTION] The comment for OrigText is clear and accurately describes its purpose and how it differs from Text. This distinction is crucial for understanding the type-1 vs. type-2 classification logic.
There was a problem hiding this comment.
Core Changes
This Pull Request introduces a significant enhancement to dupehound by classifying detected code clones into specific types (type-1 and type-2) and assigning a similarity score.
Key changes include:
domain/types.go: New constantsCloneType1,CloneType2,CloneType3are defined, and theClonestruct is extended withType(string) andSimilarity(float64) fields.services/tokenizer.go: TheTokenstruct now includes anOrigTextfield. This field stores the original source text for identifiers, numbers, and string literals, which is crucial for distinguishing clone types after the initial structural hashing. The lexer logic has been updated to populate this field correctly.services/detector.go: A newclassifyClonefunction has been added. This function compares theOrigTextof tokens across all instances of a detected clone. If allOrigTextvalues are identical, it's classified astype-1; otherwise, it'stype-2. The similarity score is currently fixed at1.0for both types.services/reporter.go: All output formats (text, JSON, SARIF) have been updated to display the new clone type and similarity information. For SARIF, newsarifResultPropertiesare introduced to embed this data.- Tests: Comprehensive unit and integration tests have been added to validate the correct classification of type-1 and type-2 clones and ensure the new fields are present and accurate in all output formats.
Concerns
- Performance/Memory Usage: The addition of the
OrigTextfield to everyTokenwill increase memory consumption, especially for large files with many tokens. WhileOrigTextis only populated forTokIdent,TokNumber, andTokString, this still represents a notable increase. TheclassifyClonefunction also adds a post-processing step that iterates through tokens for each instance, which could have a performance impact for very large clones with many instances. However, this seems like a reasonable trade-off for the added functionality.
Verdict
Approve
Code review performed by GEMINI - gemini-2.5-flash.
| // For TokKeyword and TokOperator, Text carries the verbatim value so that | ||
| // `if` ≠ `for` and `+` ≠ `-` in clone comparisons. | ||
| // OrigText preserves the original source text for Ident/Number/String tokens, | ||
| // used after detection to classify clones as type-1 vs type-2. |
There was a problem hiding this comment.
[SUGGESTION] The comment for OrigText states it's empty for Keyword/Operator. However, Text is already used for verbatim values for Keyword/Operator. It might be clearer to state that OrigText is only for Ident/Number/String and Text is only for Keyword/Operator (and empty for the others). This distinction is important for understanding the hashing logic vs. classification logic.
| return clones | ||
| } | ||
|
|
||
| // classifyClone determines the clone type and similarity by comparing |
There was a problem hiding this comment.
[SUGGESTION] The classifyClone function currently sets similarity to 1.0 for both type-1 and type-2 clones. While this aligns with the PR description for current implementation, it might be beneficial to add a comment here explicitly stating that type-3 (future work) will introduce fractional similarity scores, to avoid confusion or premature optimization attempts for type-1/type-2 in the future.
Drop redundant integration test, assert correct type values in JSON/SARIF tests instead of just checking field existence, add mixed 3-way clone test (A+B identical, C renamed → type-2). Clarify Token field comments.
There was a problem hiding this comment.
Core Changes
This Pull Request introduces a significant enhancement to dupehound by classifying detected code clones into specific types and assigning similarity scores. The main goal is to differentiate between exact duplicates (type-1) and structurally identical duplicates with differing identifiers or literals (type-2).
Key changes include:
domain/types.go: New constantsCloneType1,CloneType2,CloneType3are defined. TheClonestruct now includesType(string) andSimilarity(float64) fields to store classification results.services/tokenizer.go: TheTokenstruct is extended with anOrigTextfield. This field stores the original source text for identifiers, numbers, and string literals, allowing for post-hashing comparison to determine clone type. TheTextfield continues to store verbatim values for keywords and operators for structural hashing.services/detector.go: A newclassifyClonefunction is implemented. This function compares theOrigTextof tokens within detected clone instances. If allOrigTextvalues are identical across instances, it's classified astype-1; otherwise, it'stype-2. TheSimilarityis currently hardcoded to1.0for both types, with fractional scores reserved for future type-3 detection.services/reporter.go: All output formats (text, JSON, SARIF) are updated to display the newTypeandSimilarityinformation. SARIF output now includescloneTypeandsimilarityproperties within its results.- Tests: Comprehensive unit tests in
services/detector_test.goand integration tests inintegration/cli_test.gohave been added to validate the clone classification logic and ensure correct output formatting across different scenarios, including identical code, renamed variables, different literals, and mixed three-way clones.
Concerns
- Performance: The addition of the
OrigTextfield to theTokenstruct will increase memory consumption per token, especially for files with many identifiers, numbers, or strings. WhileOrigTextis not used in the core hashing algorithm, the increased memory footprint could become noticeable for very large codebases or when processing many files concurrently. However, theclassifyClonefunction itself is efficient as it operates on already identified clone instances.
Verdict
Approve
Code review performed by GEMINI - gemini-2.5-flash.
| @@ -47,6 +47,7 @@ file_naming_pattern: | |||
|
|
|||
| ignore: | |||
| - ".git" | |||
There was a problem hiding this comment.
[SUGGESTION] Adding .claude to the ignore list is a good practice to prevent linting or scanning of AI-generated or temporary files, which often don't adhere to project-specific conventions and can introduce noise.
Switch hash storage from string to uint64 (~5x less memory per window). Remove RawLines from TokenizedFile — preview lines are now loaded lazily from disk only for files that appear in detected clones. This keeps memory proportional to token count, not source content size.
There was a problem hiding this comment.
Core Changes
This Pull Request introduces significant enhancements to dupehound by implementing clone type classification (type-1 and type-2) and similarity scoring. The core changes involve:
- Clone Type Classification: Each detected clone is now categorized as
type-1(token-for-token identical) ortype-2(structurally identical but with differing identifiers or literals).type-3is reserved for future enhancements. - Similarity Scoring: A
similarityfield (currently1.0for both type-1 and type-2) is added to clone reports. - Memory Optimization: The
TokenizedFilestruct no longer storesRawLinesin memory. Instead, preview lines are loaded lazily from disk usingos.ReadFilewhen a clone is detected, significantly reducing memory footprint for large codebases. - Performance Improvement: The internal hashing mechanism has been optimized by switching from
stringhashes touint64hashes, reducing string allocations and improving hash comparison speed. - Output Format Updates: All output formats (text, JSON, SARIF) have been updated to include the new
typeandsimilarityfields. - Enhanced Tokenization: The
Tokenstruct inservices/tokenizer.gonow includes anOrigTextfield to store the original source text for identifiers, numbers, and strings, enabling post-hashing classification of clone types.
Concerns
No critical concerns were identified. The changes introduce valuable features and performance/memory optimizations without apparent regressions or security vulnerabilities.
Verdict
Approve
Code review performed by GEMINI - gemini-2.5-flash.
Summary
OrigText) for post-detection type classification without affecting hash-based detectionOutput examples
Text:
Clone #1 type-2 similarity: 1.00 7 lines · 34 tokens · 2 instancesJSON:
"type": "type-2", "similarity": 1.0fields on each cloneSARIF:
properties.cloneTypeandproperties.similarityon each resultTests
services/detector_test.go: type-1 classification, type-2 for renamed vars, type-2 for different literals, type-2 for same-file with different func names, all clones have non-empty typeintegration/cli_test.go: text output contains type label, type-1 for identical code, type-2 for renamed variable, JSON has type+similarity, SARIF has propertiesChecklist
lefthookpre-commit passesCloses #3