Skip to content

Commit 3688434

Browse files
fredbiclaude
andcommitted
feat(scanner): load and type-check packages without a Go toolchain
packages.Load resolves a package graph by running `go list`. That binds a scan to an installed toolchain and rules out any environment without one. internal/packages does the same job in pure Go: patterns to directories, build-constraint selection through go/build, parse, type-check. Both ways of resolving a graph live behind one Loader and are chosen by a Strategy, so the options mean the same thing under either and a caller swaps by asking, not by rewriting. Across codescan's fixture corpus the two produce identical specs. Experimental, and opt-in: ToolchainFreeLoader selects it, Options.FS implies it, and saying nothing keeps the go/packages path exactly as it was. New options: - ToolchainFreeLoader — scan without the go command. - GOOS/GOARCH/GOFLAGS/GOWORK/GOEXPERIMENT — the go environment that decides what is built, stated rather than inherited so a scan is reproducible. Honoured by both strategies; the go command reads them from the environment and nowhere else, so a pinned value is pushed there. - FS — read source through an io/fs, which is what lets a scan cover a tree that was never written to disk. - StubStdlib — synthesize the standard library instead of reading GOROOT. Trades fidelity for reach: a synthesized type has no fields and no method set. Every fabricated import raises scan.synthesized-import. - ExportData — serve dependencies from pre-computed export data instead. Resolution follows the go command rather than approximating it. `...` stops at a module boundary and is a wildcard anywhere in a pattern; import paths come from the module that contains the directory; go.work places a sibling module at its working copy rather than at a stale cached one; vendoring is decided by vendor/modules.txt as `go mod vendor` writes it; a bare pattern is an import path. A malformed directory is reported on its package and the rest of the tree still loads, as `go list -e` does. Two differences are kept on purpose, both in the direction of documenting code that compiles for its author: a tree with no module still scans, and another module's internal packages are readable. The pattern matcher itself is copied from the Go distribution under its own licence rather than re-derived — the vendor rules are subtle enough that our own version failed three of the cases upstream tests for. See NOTICE. What the copy costs is drift, so hack/go-loader is the answer to both halves of that: `sync` reports when an upstream declaration has moved, and `corpus` replays the go command's own test trees through both strategies and records each verdict in ledger.md, which is committed so a newer Go shows up as a diff. It needs a golang/go checkout and does not run in CI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Frederic BIDON <fredbi@yahoo.com>
1 parent 5c04b10 commit 3688434

46 files changed

Lines changed: 5618 additions & 44 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.claude/CLAUDE.md‎

Lines changed: 63 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,13 +34,52 @@ to builders without direct coupling.
3434
| File | Contents |
3535
|------|----------|
3636
| `options.go` | `Options` struct: packages, work dir, build tags, include/exclude, feature flags |
37-
| `scan_context.go` | `ScanCtx` / `NewScanCtx` — loads Go packages via `golang.org/x/tools/go/packages` |
37+
| `scan_context.go` | `ScanCtx` / `NewScanCtx`; `loadPackages` picks a loader from `Options.FS` (see `README.md#loader`) |
38+
| `load_golist.go` / `load_wasm.go` | the `golang.org/x/tools/go/packages` (`go list`) loader; excluded from `wasm` builds, which have no process model |
3839
| `index.go` | `TypeIndex` — node classification (meta/route/operation/model/parameters/response) |
3940
| `declaration.go` | `EntityDecl` — wraps a type/value declaration with its enclosing file/package |
4041
| `enum_value.go` | `enumBasicLitValue` — converts a `const Foo Kind = "bar"` RHS into its runtime value (enum discovery) |
4142
| `provenance.go` | `Provenance` — ties a spec JSON pointer to the source position of the Go construct that produced it; emitted via `Options.OnProvenance` (cross-ref linker, source side) |
4243
| `classify/` | Classification predicates usable from both scanner and builders (e.g. `IsAllowedExtension`) |
4344

45+
### `internal/packages/` — toolchain-free package loader (experimental)
46+
47+
Owns **both** ways of resolving a package graph, behind one `Loader`: delegate to
48+
`golang.org/x/tools/go/packages` (default), or do the same job in pure Go with no `go list` and no
49+
`exec`. The scanner states a preference and the loader reconciles it — see
50+
`internal/scanner/README.md#loader`. Keeping the switch here is what makes options like `WithTarget`
51+
mean the same thing under both. Types (`Config`, `Package`, `Error`, `LoadMode`) are aliases of the
52+
upstream ones, so the two are interchangeable at the call site.
53+
54+
Split three ways, because the halves are inherited from different places and age differently: the
55+
loader is a simplified `go/packages` (a shape we own), while `list/` is `cmd/go` (behaviour we must
56+
reproduce exactly). Quarantining the quirks is what makes them checkable against upstream.
57+
58+
| File | Contents |
59+
|------|----------|
60+
| `strategy.go` | `Strategy` (`StrategyGoPackages` / `StrategyToolchainFree`), `WithStrategy`, the `WithFS` override, shared `loadMode` |
61+
| `golist.go` / `golist_wasm.go` | the go/packages strategy + pinned-env propagation; refused on `wasm`, which has no process model |
62+
| `loader.go` | `Loader` / `NewLoader` / `Load` (dispatch); `loadFromSource`, `buildContext` (GOOS/GOARCH/cgo tiering), parse + type-check |
63+
| `goenv.go` | `GoEnv` — the pinned go environment (GOOS/GOARCH/GOFLAGS/GOWORK/GOEXPERIMENT), applied to both strategies |
64+
| `synthesize.go` | fabricates a package from the names selected through it when its source cannot be read (incl. the cgo `"C"` pseudo-package) |
65+
| `exportdata.go` | serves dependencies from pre-computed export data (`Options.ExportData`) |
66+
| `options.go` / `aliases.go` | functional options; type aliases of the upstream vocabulary |
67+
68+
#### `internal/packages/list/` — `go list` semantics
69+
70+
| File | Contents |
71+
|------|----------|
72+
| `resolve.go` | `Resolver`: patterns → directories, imports → directories; module-bounded `...` walk, nearest-module import paths, vendor mode, module cache, `replace` |
73+
| `workspace.go` | `go.work` discovery + `use`/`replace`, so a sibling module resolves to its working copy |
74+
| `pattern.go` | where a `...` walk starts, and which walked directories a pattern matches |
75+
| `pkgpattern.go` | the wildcard matcher, **copied verbatim** from the Go distribution (BSD-3-Clause; see `NOTICE`) |
76+
77+
#### `internal/packages/vfs/` — the filesystem seam
78+
79+
`vfs.FS` — the one place every read goes through (build-constraint matching, directory walking, source
80+
reading), so a virtual tree is honoured everywhere. Its own package because both halves need it and
81+
neither owns it.
82+
4483
### `internal/parsers/` — scanner classification + helpers
4584

4685
Post grammar-migration (P6.3), `parsers/` is intentionally scanner-only. The
@@ -127,6 +166,29 @@ malformed input, the petstore, aliased schemas, go123-specific forms, and cross-
127166
decisions"). See `internal/scanner/README.md#prune`.
128167
- `InputSpec` — overlay: merge discoveries on top of an existing spec.
129168
- `BuildTags`, `Include`/`Exclude`, `IncludeTags`/`ExcludeTags`, `ExcludeDeps` — scope control.
169+
- `GOOS`/`GOARCH`/`GOFLAGS`/`GOWORK`/`GOEXPERIMENT` — the go environment that
170+
decides *what* is built. Each changes the emitted spec just as `BuildTags`
171+
does, and each is an option rather than inherited state so a scan is
172+
reproducible. Empty means "whatever the environment says". Honoured by both
173+
loaders. `GOWORK` matters most: inside a workspace a sibling module resolves
174+
to its working copy, and missing it synthesizes the sibling empty rather than
175+
failing. See `internal/scanner/README.md#loader`.
176+
- Toolchain-free loading (experimental; see `internal/scanner/README.md#loader`):
177+
- `ToolchainFreeLoader` — scan through `internal/packages` instead of
178+
`golang.org/x/tools/go/packages`. Same job, same output; it just needs no
179+
installed toolchain and no `exec`, since it never runs `go list`. False
180+
(default) keeps the historic behaviour.
181+
- `FS` — read source through an `io/fs` filesystem instead of the real one.
182+
Implies `ToolchainFreeLoader`: `go list` runs against the real filesystem,
183+
so it could not honour `FS` even if asked.
184+
- `StubStdlib` — synthesize the stdlib from the names selected through it
185+
rather than reading GOROOT. Trades fidelity for reach: a synthesized type
186+
has no fields and no method set, so `json.RawMessage` stops rendering as a
187+
byte array and nothing is seen to implement `encoding.TextMarshaler`. Each
188+
fabricated import raises `scan.synthesized-import`.
189+
- `ExportData` — serve dependencies from pre-computed export data. Costs no
190+
fidelity, but is valid only for the toolchain that produced it; uncovered
191+
packages fall back to source, then to synthesis.
130192
- `RefAliases`, `TransparentAliases`, `DescWithRef` — alias handling knobs
131193
(`DescWithRef` is deprecated; see `EmitRefSiblings`).
132194
- `$ref`-sibling rendering (see `internal/builders/schema/README.md#ref-override`):

‎.golangci.yml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,16 @@ linters:
5555
- third_party$
5656
- builtin$
5757
- examples$
58+
# Copied verbatim from the Go distribution (BSD-3-Clause) so it stays
59+
# diffable against upstream. Reformatting it to our house style would
60+
# make every future re-sync a manual merge.
61+
- internal/packages/list/pkgpattern.*\.go$
5862
rules:
63+
# hack/ holds developer tools, not library code: printing a report to
64+
# stdout is what they are for.
65+
- path: hack/
66+
linters:
67+
- forbidigo
5968
# cmd/genspec-tui is a bubbletea front-end. These four disagree with the
6069
# shape of TUI code rather than with the code itself, and each fires often
6170
# enough (102 between them) that //nolint would be the scattering this
@@ -86,6 +95,8 @@ formatters:
8695
- third_party$
8796
- builtin$
8897
- examples$
98+
# As above: derived from the Go distribution, kept byte-comparable.
99+
- internal/packages/list/pkgpattern.*\.go$
89100
issues:
90101
# Maximum issues count per one linter.
91102
# Set to 0 to disable.

‎LICENSE-BSD-go‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
Copyright 2009 The Go Authors.
2+
3+
Redistribution and use in source and binary forms, with or without
4+
modification, are permitted provided that the following conditions are
5+
met:
6+
7+
* Redistributions of source code must retain the above copyright
8+
notice, this list of conditions and the following disclaimer.
9+
* Redistributions in binary form must reproduce the above
10+
copyright notice, this list of conditions and the following disclaimer
11+
in the documentation and/or other materials provided with the
12+
distribution.
13+
* Neither the name of Google LLC nor the names of its
14+
contributors may be used to endorse or promote products derived from
15+
this software without specific prior written permission.
16+
17+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
18+
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
19+
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
20+
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
21+
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
22+
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
23+
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
24+
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
25+
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
26+
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
27+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

‎NOTICE‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
Copyright 2015-2025 go-swagger maintainers
2+
3+
// SPDX-FileCopyrightText: Copyright 2015-2025 go-swagger maintainers
4+
// SPDX-License-Identifier: Apache-2.0
5+
6+
This software library, github.com/go-openapi/codescan, includes software developed
7+
by the go-swagger and go-openapi maintainers ("go-swagger maintainers").
8+
9+
Licensed under the Apache License, Version 2.0 (the "License");
10+
you may not use this software except in compliance with the License.
11+
12+
You may obtain a copy of the License at
13+
14+
http://www.apache.org/licenses/LICENSE-2.0.
15+
16+
This software is copied from, derived from, and inspired by other original software products.
17+
It ships with copies of other software which license terms are recalled below.
18+
19+
20+
The Go Programming Language
21+
===========================
22+
23+
// SPDX-FileCopyrightText: Copyright 2009 The Go Authors
24+
// SPDX-License-Identifier: BSD-3-Clause
25+
26+
The package pattern matcher in internal/packages/list/pkgpattern.go, and the test table that
27+
specifies it in internal/packages/list/pkgpattern_test.go, are copied from the Go distribution
28+
(cmd/internal/pkgpattern, go1.26.5) and carry their original copyright notice.
29+
30+
A verbatim copy of the license below also ships as LICENSE-BSD-go.
31+
32+
Copyright 2009 The Go Authors.
33+
34+
Redistribution and use in source and binary forms, with or without
35+
modification, are permitted provided that the following conditions are
36+
met:
37+
38+
* Redistributions of source code must retain the above copyright
39+
notice, this list of conditions and the following disclaimer.
40+
* Redistributions in binary form must reproduce the above
41+
copyright notice, this list of conditions and the following disclaimer
42+
in the documentation and/or other materials provided with the
43+
distribution.
44+
* Neither the name of Google LLC nor the names of its
45+
contributors may be used to endorse or promote products derived from
46+
this software without specific prior written permission.
47+
48+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
49+
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
50+
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
51+
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
52+
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
53+
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
54+
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
55+
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
56+
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
57+
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
58+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

‎cmd/genspec-tui/internal/ux/options/options.go‎

Lines changed: 18 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -26,14 +26,15 @@ import (
2626
// These are the same divisions the docs already uses to describe the knobs, so the overlay and the docs agree on where
2727
// a setting lives.
2828
const (
29-
groupScope = "discovery & scope"
30-
groupRefs = "$ref & composition"
31-
groupNaming = "naming"
32-
groupDocs = "docs & comments"
33-
groupTypes = "types & extensions"
29+
groupScope = "discovery & scope"
30+
groupRefs = "$ref & composition"
31+
groupNaming = "naming"
32+
groupDocs = "docs & comments"
33+
groupTypes = "types & extensions"
34+
groupLoading = "package loading"
3435

3536
// groupCount is how many of the above appear, for sizing only.
36-
groupCount = 5
37+
groupCount = 6
3738
)
3839

3940
// dep records that a toggle only bites when another one holds a particular value.
@@ -140,6 +141,17 @@ func New(cfg *codescan.Options) Overlay {
140141
groupTypes, "EmitXGoType", "stamp x-go-type on definitions", &cfg.EmitXGoType,
141142
&dep{depSkipExtensions, false, "SkipExtensions"},
142143
},
144+
145+
// Package loading. Last because it is experimental and rarely touched: the cursor opens on the
146+
// row people actually came for.
147+
{
148+
groupLoading, "ToolchainFreeLoader", "load packages without the go command",
149+
&cfg.ToolchainFreeLoader, nil,
150+
},
151+
{
152+
groupLoading, "StubStdlib", "synthesize the stdlib, skip GOROOT",
153+
&cfg.StubStdlib, &dep{&cfg.ToolchainFreeLoader, true, "ToolchainFreeLoader"},
154+
},
143155
},
144156
}
145157
}

‎cmd/genspec-tui/main.go‎

Lines changed: 48 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -36,16 +36,23 @@ import (
3636
type cliFlags struct {
3737
set *flag.FlagSet
3838

39-
workdir *string
40-
packages *string
41-
scanModels *bool
42-
buildTags *string
43-
include *string
44-
exclude *string
45-
includeTags *string
46-
excludeTags *string
47-
nameFromTags *string
48-
nameConcatBudget *float64
39+
workdir *string
40+
packages *string
41+
scanModels *bool
42+
buildTags *string
43+
goos *string
44+
goarch *string
45+
goflags *string
46+
gowork *string
47+
goexperiment *string
48+
toolchainFreeLoader *bool
49+
stubStdlib *bool
50+
include *string
51+
exclude *string
52+
includeTags *string
53+
excludeTags *string
54+
nameFromTags *string
55+
nameConcatBudget *float64
4956
}
5057

5158
// registerFlags declares every flag on fs.
@@ -59,8 +66,20 @@ func registerFlags(fs *flag.FlagSet) *cliFlags {
5966
packages: fs.String("packages", "./...", "comma-separated package patterns to scan, relative to -workdir"),
6067
scanModels: fs.Bool("scan-models", true, "also emit definitions for swagger:model types"),
6168
buildTags: fs.String("build-tags", "", "comma-separated go build tags to apply while loading"),
62-
include: fs.String("include", "", "comma-separated patterns; only matching packages are scanned"),
63-
exclude: fs.String("exclude", "", "comma-separated patterns; matching packages are skipped"),
69+
goos: fs.String("goos", "", "GOOS the scanned code is built for (default: this machine's)"),
70+
goarch: fs.String("goarch", "", "GOARCH the scanned code is built for (default: this machine's)"),
71+
goflags: fs.String("goflags", "",
72+
"default go command flags, as GOFLAGS (e.g. \"-tags=integration\"); -build-tags wins"),
73+
gowork: fs.String("gowork", "",
74+
`workspace selection, as GOWORK: "off" to disable, a path to a go.work, empty to search upwards`),
75+
goexperiment: fs.String("goexperiment", "",
76+
"toolchain experiments to enable, as GOEXPERIMENT (e.g. \"jsonv2\")"),
77+
toolchainFreeLoader: fs.Bool("toolchain-free-loader", false,
78+
"load packages with codescan's own loader instead of the go command (experimental)"),
79+
stubStdlib: fs.Bool("stub-stdlib", false,
80+
"synthesize the standard library instead of reading GOROOT (needs -toolchain-free-loader)"),
81+
include: fs.String("include", "", "comma-separated patterns; only matching packages are scanned"),
82+
exclude: fs.String("exclude", "", "comma-separated patterns; matching packages are skipped"),
6483
includeTags: fs.String("include-tags", "",
6584
"comma-separated swagger tags; only matching operations are emitted"),
6685
excludeTags: fs.String("exclude-tags", "",
@@ -75,16 +94,23 @@ func registerFlags(fs *flag.FlagSet) *cliFlags {
7594
// options assembles the scan config. workDir is passed in already absolute.
7695
func (c *cliFlags) options(workDir string) codescan.Options {
7796
return codescan.Options{
78-
WorkDir: workDir,
79-
Packages: splitPatterns(*c.packages),
80-
ScanModels: *c.scanModels,
81-
BuildTags: *c.buildTags,
82-
Include: splitList(*c.include),
83-
Exclude: splitList(*c.exclude),
84-
IncludeTags: splitList(*c.includeTags),
85-
ExcludeTags: splitList(*c.excludeTags),
86-
NameFromTags: resolveNameFromTags(*c.nameFromTags, c.passed("name-from-tags")),
87-
NameConcatBudget: *c.nameConcatBudget,
97+
WorkDir: workDir,
98+
Packages: splitPatterns(*c.packages),
99+
ScanModels: *c.scanModels,
100+
BuildTags: *c.buildTags,
101+
GOOS: *c.goos,
102+
GOARCH: *c.goarch,
103+
GOFLAGS: *c.goflags,
104+
GOWORK: *c.gowork,
105+
GOEXPERIMENT: *c.goexperiment,
106+
ToolchainFreeLoader: *c.toolchainFreeLoader,
107+
StubStdlib: *c.stubStdlib,
108+
Include: splitList(*c.include),
109+
Exclude: splitList(*c.exclude),
110+
IncludeTags: splitList(*c.includeTags),
111+
ExcludeTags: splitList(*c.excludeTags),
112+
NameFromTags: resolveNameFromTags(*c.nameFromTags, c.passed("name-from-tags")),
113+
NameConcatBudget: *c.nameConcatBudget,
88114
}
89115
}
90116

‎cmd/genspec-tui/main_test.go‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,11 @@ var optionFlags = map[string]string{ //nolint:gochecknoglobals // table for the
2222
"WorkDir": "workdir",
2323
"Packages": "packages",
2424
"BuildTags": "build-tags",
25+
"GOOS": "goos",
26+
"GOARCH": "goarch",
27+
"GOFLAGS": "goflags",
28+
"GOWORK": "gowork",
29+
"GOEXPERIMENT": "goexperiment",
2530
"Include": "include",
2631
"Exclude": "exclude",
2732
"IncludeTags": "include-tags",
@@ -35,6 +40,8 @@ var optionsNotOnCLI = map[string]string{ //nolint:gochecknoglobals // table for
3540
"InputSpec": "overlay mode: needs a spec loaded from disk, not yet exposed",
3641
"OnDiagnostic": "wired internally to the diagnostics pane",
3742
"OnProvenance": "wired internally to the cross-ref linker",
43+
"FS": "virtual source filesystem: a programmatic seam, not expressible on a command line",
44+
"ExportData": "a filesystem of pre-computed export data: a programmatic seam, like FS",
3845
}
3946

4047
// The CLI exposed three of ten options for a long time, because nothing failed when a new one landed.

‎go.mod‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ require (
1010
github.com/go-openapi/swag/yamlutils v0.28.0
1111
github.com/go-openapi/testify/v2 v2.6.0
1212
go.yaml.in/yaml/v3 v3.0.5
13+
golang.org/x/mod v0.38.0
1314
golang.org/x/tools v0.48.0
1415
)
1516

@@ -22,6 +23,5 @@ require (
2223
github.com/go-openapi/swag/pools v0.28.0 // indirect
2324
github.com/go-openapi/swag/stringutils v0.28.0 // indirect
2425
github.com/go-openapi/swag/typeutils v0.28.0 // indirect
25-
golang.org/x/mod v0.38.0 // indirect
2626
golang.org/x/sync v0.22.0 // indirect
2727
)

0 commit comments

Comments
 (0)
⌂ 换网址