Skip to content

feat(#3): clone type labels and similarity scores - #9

Merged
machado144 merged 4 commits into
mainfrom
feat/3-clone-type-labels
Apr 2, 2026
Merged

machado144 merged 4 commits into
mainfrom
feat/3-clone-type-labels

Conversation

@machado144

Copy link
Copy Markdown
Contributor

Summary

  • Classifies each detected clone as type-1 (token-for-token identical) or type-2 (same structure, differing identifiers/literals)
  • Adds similarity score (1.0 for both type-1 and type-2; type-3 with fractional scores reserved for feat: type-3 fuzzy clone detection (near-miss blocks) #4)
  • Preserves original token text (OrigText) for post-detection type classification without affecting hash-based detection
  • Updates all three output formats: text, JSON, and SARIF

Output examples

Text: Clone #1 type-2 similarity: 1.00 7 lines · 34 tokens · 2 instances

JSON: "type": "type-2", "similarity": 1.0 fields on each clone

SARIF: properties.cloneType and properties.similarity on each result

Tests

  • 5 unit tests in services/detector_test.go: type-1 classification, type-2 for renamed vars, type-2 for different literals, type-2 for same-file with different func names, all clones have non-empty type
  • 5 integration tests in integration/cli_test.go: text output contains type label, type-1 for identical code, type-2 for renamed variable, JSON has type+similarity, SARIF has properties

Checklist

  • All tests pass (117 tests, 0 regressions)
  • lefthook pre-commit passes
  • No regressions in existing 107 tests

Closes #3

Classify each detected clone as type-1 (token-for-token identical) or
type-2 (same structure, differing identifiers/literals) with a similarity
score. Adds OrigText to Token for post-detection comparison, updates text,
JSON, and SARIF formatters, and includes 10 new tests (5 unit + 5 integration).
@github-actions

github-actions Bot commented Apr 2, 2026 •

Copy link
Copy Markdown

StructLint — All checks passed

46 rules validated against .structlint.yaml. No violations found.

View full run · Powered by StructLint

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This Pull Request introduces the classification of detected code clones into 'type-1' and 'type-2', along with a similarity score.

Key changes include:

  • Data Model Enhancement: The domain.Clone struct now includes Type (string) and Similarity (float64) fields, with constants defined for CloneType1, CloneType2, and CloneType3.
  • Lexer Update: The services/tokenizer.go now stores the original source text (OrigText) for TokIdent, TokNumber, and TokString tokens. This OrigText is crucial for post-detection classification without affecting the structural hashing.
  • Clone Classification Logic: A new classifyClone function in services/detector.go compares the OrigText of tokens across clone instances to determine if they are type-1 (token-for-token identical) or type-2 (structurally identical but with differing identifiers/literals). For both type-1 and type-2, the similarity is currently set to 1.0.
  • Output Format Updates: All output formats (text, JSON, SARIF) have been updated to display the new clone type and similarity score.
  • Comprehensive Testing: New unit tests in services/detector_test.go and integration tests in integration/cli_test.go validate the correctness of clone type classification and its representation in various outputs.

Concerns (if any)

None. The changes are well-isolated, thoroughly tested, and align with the project's goals for clone detection.

Verdict

Approve


Code review performed by GEMINI - gemini-2.5-flash.

Comment thread domain/types.go
Lines []string `json:"lines"`
}

// Clone type constants.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] The CloneType3 constant is defined but the PR summary mentions it's reserved for future work (#4). It's good to have it defined early, but ensure that the current implementation doesn't accidentally report type-3 clones or misrepresent their similarity if the fuzzy detector isn't yet integrated. The current classifyClone function correctly defaults to type-1 or type-2.

Comment thread services/detector.go
@@ -222,6 +226,31 @@ func Detect(files []TokenizedFile, minTokens int) []domain.Clone {
return clones
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] The classifyClone function includes a check if len(starts) < 2 { return domain.CloneType1, 1.0 }. This condition is redundant because the Detect function already ensures that len(starts) is at least 2 before calling classifyClone (lines 179-181). While harmless, it could be removed for cleaner code.

Comment thread services/tokenizer.go
// `if` ≠ `for` and `+` ≠ `-` in clone comparisons.
// OrigText preserves the original source text for Ident/Number/String tokens,
// used after detection to classify clones as type-1 vs type-2.
type Token struct {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] The comment for OrigText is clear and accurately describes its purpose and how it differs from Text. This distinction is crucial for understanding the type-1 vs. type-2 classification logic.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This Pull Request introduces a significant enhancement to dupehound by classifying detected code clones into specific types (type-1 and type-2) and assigning a similarity score.

Key changes include:

  • domain/types.go: New constants CloneType1, CloneType2, CloneType3 are defined, and the Clone struct is extended with Type (string) and Similarity (float64) fields.
  • services/tokenizer.go: The Token struct now includes an OrigText field. This field stores the original source text for identifiers, numbers, and string literals, which is crucial for distinguishing clone types after the initial structural hashing. The lexer logic has been updated to populate this field correctly.
  • services/detector.go: A new classifyClone function has been added. This function compares the OrigText of tokens across all instances of a detected clone. If all OrigText values are identical, it's classified as type-1; otherwise, it's type-2. The similarity score is currently fixed at 1.0 for both types.
  • services/reporter.go: All output formats (text, JSON, SARIF) have been updated to display the new clone type and similarity information. For SARIF, new sarifResultProperties are introduced to embed this data.
  • Tests: Comprehensive unit and integration tests have been added to validate the correct classification of type-1 and type-2 clones and ensure the new fields are present and accurate in all output formats.

Concerns

  • Performance/Memory Usage: The addition of the OrigText field to every Token will increase memory consumption, especially for large files with many tokens. While OrigText is only populated for TokIdent, TokNumber, and TokString, this still represents a notable increase. The classifyClone function also adds a post-processing step that iterates through tokens for each instance, which could have a performance impact for very large clones with many instances. However, this seems like a reasonable trade-off for the added functionality.

Verdict

Approve


Code review performed by GEMINI - gemini-2.5-flash.

Comment thread services/tokenizer.go Outdated
// For TokKeyword and TokOperator, Text carries the verbatim value so that
// `if` ≠ `for` and `+` ≠ `-` in clone comparisons.
// OrigText preserves the original source text for Ident/Number/String tokens,
// used after detection to classify clones as type-1 vs type-2.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] The comment for OrigText states it's empty for Keyword/Operator. However, Text is already used for verbatim values for Keyword/Operator. It might be clearer to state that OrigText is only for Ident/Number/String and Text is only for Keyword/Operator (and empty for the others). This distinction is important for understanding the hashing logic vs. classification logic.

Comment thread services/detector.go
return clones
}

// classifyClone determines the clone type and similarity by comparing

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] The classifyClone function currently sets similarity to 1.0 for both type-1 and type-2 clones. While this aligns with the PR description for current implementation, it might be beneficial to add a comment here explicitly stating that type-3 (future work) will introduce fractional similarity scores, to avoid confusion or premature optimization attempts for type-1/type-2 in the future.

Drop redundant integration test, assert correct type values in JSON/SARIF
tests instead of just checking field existence, add mixed 3-way clone test
(A+B identical, C renamed → type-2). Clarify Token field comments.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This Pull Request introduces a significant enhancement to dupehound by classifying detected code clones into specific types and assigning similarity scores. The main goal is to differentiate between exact duplicates (type-1) and structurally identical duplicates with differing identifiers or literals (type-2).

Key changes include:

  • domain/types.go: New constants CloneType1, CloneType2, CloneType3 are defined. The Clone struct now includes Type (string) and Similarity (float64) fields to store classification results.
  • services/tokenizer.go: The Token struct is extended with an OrigText field. This field stores the original source text for identifiers, numbers, and string literals, allowing for post-hashing comparison to determine clone type. The Text field continues to store verbatim values for keywords and operators for structural hashing.
  • services/detector.go: A new classifyClone function is implemented. This function compares the OrigText of tokens within detected clone instances. If all OrigText values are identical across instances, it's classified as type-1; otherwise, it's type-2. The Similarity is currently hardcoded to 1.0 for both types, with fractional scores reserved for future type-3 detection.
  • services/reporter.go: All output formats (text, JSON, SARIF) are updated to display the new Type and Similarity information. SARIF output now includes cloneType and similarity properties within its results.
  • Tests: Comprehensive unit tests in services/detector_test.go and integration tests in integration/cli_test.go have been added to validate the clone classification logic and ensure correct output formatting across different scenarios, including identical code, renamed variables, different literals, and mixed three-way clones.

Concerns

  • Performance: The addition of the OrigText field to the Token struct will increase memory consumption per token, especially for files with many identifiers, numbers, or strings. While OrigText is not used in the core hashing algorithm, the increased memory footprint could become noticeable for very large codebases or when processing many files concurrently. However, the classifyClone function itself is efficient as it operates on already identified clone instances.

Verdict

Approve


Code review performed by GEMINI - gemini-2.5-flash.

Comment thread .structlint.yaml
@@ -47,6 +47,7 @@ file_naming_pattern:

ignore:
- ".git"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION] Adding .claude to the ignore list is a good practice to prevent linting or scanning of AI-generated or temporary files, which often don't adhere to project-specific conventions and can introduce noise.

Switch hash storage from string to uint64 (~5x less memory per window).
Remove RawLines from TokenizedFile — preview lines are now loaded lazily
from disk only for files that appear in detected clones. This keeps
memory proportional to token count, not source content size.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This Pull Request introduces significant enhancements to dupehound by implementing clone type classification (type-1 and type-2) and similarity scoring. The core changes involve:

  1. Clone Type Classification: Each detected clone is now categorized as type-1 (token-for-token identical) or type-2 (structurally identical but with differing identifiers or literals). type-3 is reserved for future enhancements.
  2. Similarity Scoring: A similarity field (currently 1.0 for both type-1 and type-2) is added to clone reports.
  3. Memory Optimization: The TokenizedFile struct no longer stores RawLines in memory. Instead, preview lines are loaded lazily from disk using os.ReadFile when a clone is detected, significantly reducing memory footprint for large codebases.
  4. Performance Improvement: The internal hashing mechanism has been optimized by switching from string hashes to uint64 hashes, reducing string allocations and improving hash comparison speed.
  5. Output Format Updates: All output formats (text, JSON, SARIF) have been updated to include the new type and similarity fields.
  6. Enhanced Tokenization: The Token struct in services/tokenizer.go now includes an OrigText field to store the original source text for identifiers, numbers, and strings, enabling post-hashing classification of clone types.

Concerns

No critical concerns were identified. The changes introduce valuable features and performance/memory optimizations without apparent regressions or security vulnerabilities.

Verdict

Approve


Code review performed by GEMINI - gemini-2.5-flash.

@machado144
machado144 merged commit 30d7f51 into main Apr 2, 2026
6 checks passed
@machado144
machado144 deleted the feat/3-clone-type-labels branch April 2, 2026 11:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: clone type labels and similarity scores in all output formats

1 participant

⌂ 换网址