Skip to content

chore: move test-only BuildTokenizedFile and Detect into _test.go - #25

Merged
machado144 merged 2 commits into
mainfrom
cleanup/dead-test-helpers
Apr 14, 2026
Merged

machado144 merged 2 commits into
mainfrom
cleanup/dead-test-helpers

Conversation

@machado144

@machado144 machado144 commented Apr 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Acts on the dead code and duplication that dupehound scan --dead-code flagged when run against its own source.

Dead test-only exports

  • BuildTokenizedFile and Detect in services/detector.go had zero production callers — only referenced from _test.go files. Both are thin wrappers over the real production functions (BuildTokenizedFileWithIgnore, DetectWithOptions).
  • Moved into services/testhelpers_test.go so they compile only for tests. No test call sites change.

Duplication cleanup (acting on what the tool reported)

  • services/reporter.go: extract partitionClones and per-format writeClonesTextSection / writeClonesMdSection helpers. Test-prod and normal-clone sections in both formatters now share the partitioning and the capped-loop-plus-overflow-line logic.
  • services/deadcode.go: extract appendFuncDef so every per-language scanner in extractFunctionDefs records definitions through one struct-literal site.
  • services/detector.go: fold hashWindow and hashWindowFull onto a shared hashTokens core (the only real difference was mixing in OrigText).

Skipped the scan.go c.IsSet override boilerplate — idiomatic urfave/cli; extracting it would be noisier than the duplication.

Self-scan impact: 17 → 10 clones, 8.6% → 6.2% duplication. Remaining clones live in text-vs-markdown preamble / hotspot tables where the output shapes genuinely differ (plain lines vs markdown tables).

Test plan

  • go build ./... passes
  • go test ./... — 274 tests pass across 6 packages
  • dupehound scan --dead-code no longer reports the two functions
  • dupehound scan reports fewer clones and lower duplication %
  • lefthook pre-commit (gofumpt, structlint, govet, golangci-lint) clean

dupehound scan --dead-code on its own source flagged BuildTokenizedFile and
Detect as having no production callers — both are thin wrappers over the real
production functions (BuildTokenizedFileWithIgnore, DetectWithOptions) and are
only referenced from _test.go files.

Move both into services/testhelpers_test.go so they are compiled only for
tests. No test call sites change; production binary no longer ships them.
@github-actions

github-actions Bot commented Apr 14, 2026 •

Copy link
Copy Markdown

StructLint — All checks passed

73 rules validated against .structlint.yaml. No violations found.

View full run · Powered by StructLint

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This pull request refactors the services package by moving two test-only helper functions, BuildTokenizedFile and Detect, from services/detector.go to a new test-specific file, services/testhelpers_test.go. These functions were thin wrappers around the production functions (BuildTokenizedFileWithIgnore and DetectWithOptions) and were only ever called from test files.

This change improves code organization by making these utilities internal to the test suite, reducing the public API surface of the services package, and slightly optimizing the production binary by not compiling these unused exports.


Concerns

None. This is a beneficial refactoring that improves code clarity and reduces unnecessary exports without impacting functionality or introducing regressions.


Verdict

Approve: Changes look good and are safe to merge


Code review performed by GEMINI - gemini-2.5-flash.

…ls, fnv hashing

Acts on the clones dupehound itself flagged:

- services/reporter.go: extract partitionClones() and per-format
  writeClonesTextSection / writeClonesMdSection helpers. Test-prod and
  normal-clone sections in both formatters now share the partitioning
  and the capped-loop-plus-overflow-line logic.
- services/deadcode.go: extract appendFuncDef() so every per-language
  scanner in extractFunctionDefs records definitions through one
  struct-literal site.
- services/detector.go: fold hashWindow and hashWindowFull onto a shared
  hashTokens core; the only real difference was mixing in OrigText.

Skipped the scan.go c.IsSet override boilerplate — idiomatic urfave/cli;
extracting it would be noisier than the duplication.

Self-scan result: 17 → 10 clones, 8.6% → 6.2% duplication. Remaining
clones live in text-vs-markdown preamble / hotspot tables where the
output shapes genuinely differ (plain lines vs markdown tables), so
unifying them further would hurt readability.
@github-actions

Copy link
Copy Markdown

dupehound scan results

Scanning diff since: origin/main (4 files changed)

Metric Value
Files scanned 17 / 17
Total lines 3547
Clones found 7
Duplicate lines 183 (5.2%)
New clones since origin/main 2
Breakdown 7 type-2

Hotspots

Dup % Dup Lines Total Lines File
25.2% 157 622 services/reporter.go
11.5% 26 227 services/deadcode.go

Top clones (by impact)

#1   type-2   similarity: 1.00   27 lines   2 instances

Locations:

  • services/reporter.go:192-218
  • services/reporter.go:340-367

Preview:

			fmt.Fprintf(&b, "  ... %d more files (use --verbose to show all)\n", len(report.FileStats)-limit)
		}
	}

	if report.TotalClones > 0 {
		testProdClones, normalClones := partitionClones(sortClonesByImpact(report.Clones), opts.ShowSuppressed)

		if len(testProdClones) > 0 {
			fmt.Fprintf(&b, "\nTest↔Prod clones (span test and production files):\n")
			writeClonesTextSection(&b, testProdClones, opts, "test↔prod clones")
// ... 17 more lines
#2   type-2   similarity: 1.00   14 lines   2 instances

Locations:

  • services/reporter.go:158-171
  • services/reporter.go:294-306

Preview:

	fmt.Fprintf(&b, "Files scanned : %d / %d\n", report.ScannedFiles, report.TotalFiles)
	if report.SkippedFiles > 0 {
		fmt.Fprintf(&b, "Skipped files : %d (binary)\n", report.SkippedFiles)
	}
	fmt.Fprintf(&b, "Total lines   : %d\n", report.TotalLines)
	if report.SuppressedClones > 0 {
		fmt.Fprintf(&b, "Clones found  : %d (%d suppressed)\n", report.TotalClones, report.SuppressedClones)
	} else {
		fmt.Fprintf(&b, "Clones found  : %d\n", report.TotalClones)
	}
// ... 4 more lines
#3   type-2   similarity: 1.00   13 lines   2 instances

Locations:

  • services/deadcode.go:179-191
  • services/deadcode.go:196-208

Preview:

			if tok.Kind == TokKeyword && defKws[tok.Text] {
				for j := i + 1; j < n; j++ {
					next := tf.Tokens[j]
					if next.Kind == TokIdent && next.OrigText != "" {
						defs = appendFuncDef(defs, tf, langName, j)
						break
					}
					if next.Kind == TokOperator && next.Text == ":" {
						break
					}
// ... 3 more lines
#4   type-2   similarity: 1.00   13 lines   2 instances

Locations:

  • services/reporter.go:171-183
  • services/reporter.go:306-319

Preview:

		fmt.Fprintf(&b, "New clones    : %d since %s\n", len(report.NewClones), report.SinceDiffRef)
	}

	if report.TotalClones == 0 && len(report.DeadFunctions) == 0 {
		fmt.Fprintf(&b, "\nNo duplicates found.\n")
		return b.String()
	}

	if report.TotalClones > 0 {
		fmt.Fprintf(&b, "Breakdown     : %s\n", typeBreakdown(report.Clones))
// ... 3 more lines
#5   type-2   similarity: 1.00   10 lines   2 instances

Locations:

  • services/reporter.go:245-254
  • services/reporter.go:403-412

Preview:

		num, c.Type, c.Similarity, c.LineCount, c.TokenCount, len(c.Instances), testProdLabel, tag)
	instLimit := len(c.Instances)
	if !opts.Verbose && instLimit > maxInstancesShown {
		instLimit = maxInstancesShown
	}
	for i := 0; i < instLimit; i++ {
		inst := c.Instances[i]
		testLabel := ""
		if inst.IsTest {
			testLabel = " [test]"
#6   type-2   similarity: 1.00   9 lines   2 instances

Locations:

  • services/reporter.go:184-192
  • services/reporter.go:323-332

Preview:

		limit := topN(len(report.FileStats), defaultTopHotspots, opts.Verbose)
		fmt.Fprintf(&b, "\nHotspots (files by duplication %%):\n")
		for i := 0; i < limit; i++ {
			fs := report.FileStats[i]
			rel := relPath(fs.File, opts.ScanPath)
			fmt.Fprintf(&b, "  %5.1f%%  %4d/%4d lines  %s\n", fs.DuplicationPct, fs.DuplicateLines, fs.TotalLines, rel)
		}
		if len(report.FileStats) > limit {
			fmt.Fprintf(&b, "  ... %d more files (use --verbose to show all)\n", len(report.FileStats)-limit)
#7   type-2   similarity: 1.00   6 lines   2 instances

Locations:

  • services/reporter.go:272-277
  • services/reporter.go:431-436

Preview:

			for _, line := range lines[:limit] {
				fmt.Fprintf(b, "        | %s\n", strings.TrimRight(line, " \t"))
			}
			if len(lines) > previewMaxLines {
				fmt.Fprintf(b, "        ... (%d more lines)\n", len(lines)-previewMaxLines)
			}

View full run · Scanned by dupehound

@github-actions

Copy link
Copy Markdown

dupehound scan results

Scanning diff since: origin/main (4 files changed)

Metric Value
Files scanned 17 / 17
Total lines 3547
Clones found 7
Duplicate lines 183 (5.2%)
New clones since origin/main 2
Breakdown 7 type-2

Hotspots

Dup % Dup Lines Total Lines File
25.2% 157 622 services/reporter.go
11.5% 26 227 services/deadcode.go

Top clones (by impact)

#1   type-2   similarity: 1.00   27 lines   2 instances

Locations:

  • services/reporter.go:192-218
  • services/reporter.go:340-367

Preview:

			fmt.Fprintf(&b, "  ... %d more files (use --verbose to show all)\n", len(report.FileStats)-limit)
		}
	}

	if report.TotalClones > 0 {
		testProdClones, normalClones := partitionClones(sortClonesByImpact(report.Clones), opts.ShowSuppressed)

		if len(testProdClones) > 0 {
			fmt.Fprintf(&b, "\nTest↔Prod clones (span test and production files):\n")
			writeClonesTextSection(&b, testProdClones, opts, "test↔prod clones")
// ... 17 more lines
#2   type-2   similarity: 1.00   14 lines   2 instances

Locations:

  • services/reporter.go:158-171
  • services/reporter.go:294-306

Preview:

	fmt.Fprintf(&b, "Files scanned : %d / %d\n", report.ScannedFiles, report.TotalFiles)
	if report.SkippedFiles > 0 {
		fmt.Fprintf(&b, "Skipped files : %d (binary)\n", report.SkippedFiles)
	}
	fmt.Fprintf(&b, "Total lines   : %d\n", report.TotalLines)
	if report.SuppressedClones > 0 {
		fmt.Fprintf(&b, "Clones found  : %d (%d suppressed)\n", report.TotalClones, report.SuppressedClones)
	} else {
		fmt.Fprintf(&b, "Clones found  : %d\n", report.TotalClones)
	}
// ... 4 more lines
#3   type-2   similarity: 1.00   13 lines   2 instances

Locations:

  • services/deadcode.go:179-191
  • services/deadcode.go:196-208

Preview:

			if tok.Kind == TokKeyword && defKws[tok.Text] {
				for j := i + 1; j < n; j++ {
					next := tf.Tokens[j]
					if next.Kind == TokIdent && next.OrigText != "" {
						defs = appendFuncDef(defs, tf, langName, j)
						break
					}
					if next.Kind == TokOperator && next.Text == ":" {
						break
					}
// ... 3 more lines
#4   type-2   similarity: 1.00   13 lines   2 instances

Locations:

  • services/reporter.go:171-183
  • services/reporter.go:306-319

Preview:

		fmt.Fprintf(&b, "New clones    : %d since %s\n", len(report.NewClones), report.SinceDiffRef)
	}

	if report.TotalClones == 0 && len(report.DeadFunctions) == 0 {
		fmt.Fprintf(&b, "\nNo duplicates found.\n")
		return b.String()
	}

	if report.TotalClones > 0 {
		fmt.Fprintf(&b, "Breakdown     : %s\n", typeBreakdown(report.Clones))
// ... 3 more lines
#5   type-2   similarity: 1.00   10 lines   2 instances

Locations:

  • services/reporter.go:245-254
  • services/reporter.go:403-412

Preview:

		num, c.Type, c.Similarity, c.LineCount, c.TokenCount, len(c.Instances), testProdLabel, tag)
	instLimit := len(c.Instances)
	if !opts.Verbose && instLimit > maxInstancesShown {
		instLimit = maxInstancesShown
	}
	for i := 0; i < instLimit; i++ {
		inst := c.Instances[i]
		testLabel := ""
		if inst.IsTest {
			testLabel = " [test]"
#6   type-2   similarity: 1.00   9 lines   2 instances

Locations:

  • services/reporter.go:184-192
  • services/reporter.go:323-332

Preview:

		limit := topN(len(report.FileStats), defaultTopHotspots, opts.Verbose)
		fmt.Fprintf(&b, "\nHotspots (files by duplication %%):\n")
		for i := 0; i < limit; i++ {
			fs := report.FileStats[i]
			rel := relPath(fs.File, opts.ScanPath)
			fmt.Fprintf(&b, "  %5.1f%%  %4d/%4d lines  %s\n", fs.DuplicationPct, fs.DuplicateLines, fs.TotalLines, rel)
		}
		if len(report.FileStats) > limit {
			fmt.Fprintf(&b, "  ... %d more files (use --verbose to show all)\n", len(report.FileStats)-limit)
#7   type-2   similarity: 1.00   6 lines   2 instances

Locations:

  • services/reporter.go:272-277
  • services/reporter.go:431-436

Preview:

			for _, line := range lines[:limit] {
				fmt.Fprintf(b, "        | %s\n", strings.TrimRight(line, " \t"))
			}
			if len(lines) > previewMaxLines {
				fmt.Fprintf(b, "        ... (%d more lines)\n", len(lines)-previewMaxLines)
			}

View full run · Scanned by dupehound

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This PR focuses on significant code refactoring to reduce duplication and improve code organization, directly addressing findings from dupehound run against its own source. Key changes include:

  • Encapsulation of Test Helpers: The BuildTokenizedFile and Detect functions, previously in services/detector.go but only used by tests, have been moved to a new services/testhelpers_test.go file. This improves the clarity of the services package's public API by removing test-specific exports.
  • Refactoring Clone Reporting Logic: In services/reporter.go, duplicated logic for partitioning clones (test↔prod vs. normal), handling suppression, and formatting output with limits and overflow messages has been extracted into new helper functions: partitionClones, writeClonesTextSection, and writeClonesMdSection. This significantly cleans up formatText and formatMarkdown.
  • Consolidation of Hashing Logic: In services/detector.go, the hashWindow and hashWindowFull functions have been consolidated into a single hashTokens function, reducing redundant code by parameterizing the inclusion of OrigText.
  • Centralized Function Definition Appending: In services/deadcode.go, the appendFuncDef helper was introduced to centralize the FunctionDef struct literal creation, reducing duplication across per-language scanners in extractFunctionDefs.

These changes have resulted in a reduction of reported clones and overall duplication percentage within the dupehound codebase itself.


Concerns

None. The changes are purely refactoring and improve code quality without introducing new risks.


Verdict

Approve: Changes look good and are safe to merge


Code review performed by GEMINI - gemini-2.5-flash.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core Changes

This PR focuses on significant code refactoring to reduce duplication and improve code organization, directly addressing findings from dupehound run against its own source. Key changes include:

  • Encapsulation of Test Helpers: The BuildTokenizedFile and Detect functions, previously in services/detector.go but only used by tests, have been moved to a new services/testhelpers_test.go file. This improves the clarity of the services package's public API by removing test-specific exports.
  • Refactoring Clone Reporting Logic: In services/reporter.go, duplicated logic for partitioning clones (test↔prod vs. normal), handling suppression, and formatting output with limits and overflow messages has been extracted into new helper functions: partitionClones, writeClonesTextSection, and writeClonesMdSection. This significantly cleans up formatText and formatMarkdown.
  • Consolidation of Hashing Logic: In services/detector.go, the hashWindow and hashWindowFull functions have been consolidated into a single hashTokens function, reducing redundant code by parameterizing the inclusion of OrigText.
  • Centralized Function Definition Appending: In services/deadcode.go, the appendFuncDef helper was introduced to centralize the FunctionDef struct literal creation, reducing duplication across per-language scanners in extractFunctionDefs.

These changes have resulted in a reduction of reported clones and overall duplication percentage within the dupehound codebase itself.


Concerns

None. The changes are purely refactoring and improve code quality without introducing new risks.


Verdict

Approve: Changes look good and are safe to merge


Code review performed by GEMINI - gemini-2.5-flash.

@machado144
machado144 merged commit 43e235b into main Apr 14, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

⌂ 换网址