Skip to content

Commit 3d91dd5

Browse files
fredbiclaude
andcommitted
feat(scanner): CleanGoDoc — marker contract for idiom recomposition
Adds the marker machinery that bridges the two halves of idiom recomposition (the load-bearing design piece): which prose is godoc-derived is only known at the consumption seam, but a referenced schema's final exposed name is only known after the spec builder reduces definition names. A resolvable doc-link is encoded here as a marker carrying the schema's fully-qualified definition key; a post-reduce pass substitutes the final name. godoclink changes: - Clean now takes Options{Mangler, Resolver, Self} (was a bare *NameMangler); Resolution{DefKey, Suffix} and the Resolver callback model identifier→schema resolution (schemas only); SelfRef models the leading godoc-convention self-name. - marker encode + SubstituteMarkers decode, NUL/Unit-Separator delimited (never occur in Go comments). A marker carries (defKey, exposed field-chain suffix, humanized fallback, sentence-initial titleize bit). SubstituteMarkers maps defKey→final name (falling back to the humanized leaf for a pruned/unresolved key) and titleizes the first identifier; no marker survives the pass. - leading self-name span handling: with a Resolver + Self, the decl's own godoc-convention leading name recomposes to its exposed name (titleized); without a Resolver it is left verbatim. The live CleanGoDoc path passes a nil Resolver (resolution-free cleanup only), so output is unchanged and the integration goldens are untouched — the real go/types resolver and the post-reduce substitution land together in P3, where markers first reach live output. The marker round-trip (emit→substitute, incl. pruned-key fallback and collision rename) is unit-tested with a fake resolver. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Frederic BIDON <fredbi@yahoo.com>
1 parent e98a4eb commit 3d91dd5

6 files changed

Lines changed: 335 additions & 42 deletions

File tree

‎internal/builders/common/builder.go‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -196,7 +196,9 @@ func (s *Builder) CleanGoDoc(text string) string {
196196
return text
197197
}
198198

199-
return godoclink.Clean(text, s.Ctx.Mangler())
199+
// P1: resolution-free cleanup only. The Resolver / Self (idiom recomposition)
200+
// are wired in a later phase.
201+
return godoclink.Clean(text, godoclink.Options{Mangler: s.Ctx.Mangler()})
200202
}
201203

202204
// AppendPostDecl marks decl for post-processing by the spec

‎internal/builders/godoclink/godoclink.go‎

Lines changed: 149 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -5,17 +5,25 @@
55
// Go doc comment is carried into a Swagger title / description (the
66
// Options.CleanGoDoc feature).
77
//
8-
// Phase 1 (this package as it stands) covers the resolution-free transforms:
8+
// Two transforms apply to godoc-derived prose:
99
//
10-
// - reference-style link definition lines (`[text]: url`) are dropped;
11-
// - godoc doc-link spans (`[Widget]`, `[pkg.Type]`, `[Order.Field]`) have
12-
// their brackets removed and the (leaf) identifier humanized via the swag
13-
// name mangler — e.g. `[CustName]` → "cust name";
14-
// - the first identifier of the prose is restored to sentence case.
10+
// - resolution-free cleanup: reference-style link definition lines
11+
// (`[text]: url`) are dropped; godoc doc-link spans (`[Widget]`,
12+
// `[pkg.Type]`, `[Order.Field]`) have their brackets removed and the (leaf)
13+
// identifier humanized via the swag name mangler — e.g. `[CustName]` →
14+
// "cust name"; the first identifier of the prose is restored to sentence
15+
// case;
16+
// - idiom recomposition: when a [Resolver] maps a doc-link (or the leading
17+
// godoc-convention self-name) to an emitted schema, the span is replaced by
18+
// a [marker] carrying that schema's fully-qualified definition key. Markers
19+
// are resolved to the schema's final exposed name by [SubstituteMarkers],
20+
// run after the spec builder has reduced definition names. This two-step
21+
// dance is needed because the final name is only known at the very end of
22+
// the build, whereas which prose is godoc-derived is only known here, at the
23+
// consumption seam.
1524
//
16-
// A later phase adds identifier→schema resolution so a doc-link that points at
17-
// an emitted definition is recomposed to that definition's exposed name rather
18-
// than merely humanized. The recognizer regexes are adapted from the
25+
// With a nil [Options.Resolver] (and nil Self), only the resolution-free cleanup
26+
// runs and no marker is produced. The recognizer regexes are adapted from the
1927
// battle-tested github.com/fredbi/go-fred-mcp/pkg/doc-filters/godoc-filter; the
2028
// key difference is that this package *rewrites* the prose whereas that tool
2129
// *redacts* (length-preserving blanking) for masking.
@@ -48,17 +56,57 @@ var refDefLineRE = regexp.MustCompile(`^[ \t]*\[[^\]]+\]:[ \t]+\S.*$`)
4856
// interior reference-definition line) back to a single paragraph break.
4957
var multiNewlineRE = regexp.MustCompile(`\n{3,}`)
5058

51-
// Clean applies the resolution-free godoc filtering to text using m for
52-
// identifier humanization. It is the caller's responsibility to apply Clean
53-
// ONLY to godoc-derived prose — never to author-written swagger:title /
54-
// swagger:description override text. Clean never mutates m.
55-
func Clean(text string, m *mangling.NameMangler) string {
59+
// identRE matches a leading Go-identifier chain (the godoc-convention self-name
60+
// at the start of a doc comment, e.g. "Widget" in "Widget does things").
61+
var identRE = regexp.MustCompile(`^[\p{L}_][\p{L}\p{N}_]*`)
62+
63+
// Resolution is the outcome of resolving a doc-link reference to an emitted
64+
// schema. DefKey is the referenced type's fully-qualified definition key (whose
65+
// final exposed name is substituted later); Suffix is the already-exposed field
66+
// chain for a dotted member reference (e.g. ".customer_name"), or "" for a bare
67+
// type reference.
68+
type Resolution struct {
69+
DefKey string
70+
Suffix string
71+
}
72+
73+
// Resolver maps a doc-link reference — the bracket content with any leading `*`
74+
// stripped, e.g. "Order.CustName" or "pkg.Type" — to a [Resolution]. It returns
75+
// ok=false when the reference does not resolve to an emitted schema, in which
76+
// case the caller humanizes the leaf identifier instead.
77+
type Resolver func(ref string) (Resolution, bool)
78+
79+
// SelfRef describes the declaration whose godoc is being cleaned, so the leading
80+
// godoc-convention self-name ("Widget" in "Widget does things") can be
81+
// recomposed to the declaration's own exposed name. Name is the Go identifier;
82+
// DefKey is its fully-qualified definition key.
83+
type SelfRef struct {
84+
Name string
85+
DefKey string
86+
}
87+
88+
// Options configures Clean.
89+
type Options struct {
90+
// Mangler humanizes leaf identifiers (and marker fallbacks). Required.
91+
Mangler *mangling.NameMangler
92+
// Resolver, when non-nil, recomposes resolvable doc-links into markers;
93+
// nil selects resolution-free cleanup only.
94+
Resolver Resolver
95+
// Self, when non-nil, enables leading self-name recomposition. It has effect
96+
// only together with a non-nil Resolver.
97+
Self *SelfRef
98+
}
99+
100+
// Clean applies godoc filtering to text. It is the caller's responsibility to
101+
// apply Clean ONLY to godoc-derived prose — never to author-written
102+
// swagger:title / swagger:description override text. Clean never mutates o.
103+
func Clean(text string, o Options) string {
56104
if text == "" {
57105
return text
58106
}
59107

60108
text = dropRefDefLines(text)
61-
text = rewriteDocLinks(text, m)
109+
text = rewrite(text, o)
62110

63111
return text
64112
}
@@ -92,44 +140,107 @@ func dropRefDefLines(text string) string {
92140
return strings.TrimRight(out, " \t\n")
93141
}
94142

95-
// rewriteDocLinks strips the brackets off each doc-link span and humanizes the
96-
// leaf identifier; a span at the start of the prose is sentence-cased.
97-
func rewriteDocLinks(text string, m *mangling.NameMangler) string {
98-
locs := docLinkRE.FindAllStringIndex(text, -1)
99-
if locs == nil {
143+
// span is one stretch of text to replace: a doc-link bracket span or the
144+
// leading self-name.
145+
type span struct {
146+
lo, hi int
147+
ref string // resolver reference (full inner chain, `*` stripped); self-name = Self.Name
148+
leaf string // trailing identifier, for the humanize fallback
149+
self bool
150+
}
151+
152+
// rewrite replaces every doc-link span (and, when enabled, the leading
153+
// self-name) left to right, emitting a marker for resolvable references and the
154+
// humanized leaf otherwise.
155+
func rewrite(text string, o Options) string {
156+
spans := collectSpans(text, o)
157+
if len(spans) == 0 {
100158
return text
101159
}
102160

103161
var b strings.Builder
104162
b.Grow(len(text))
105163
last := 0
106-
for _, loc := range locs {
107-
b.WriteString(text[last:loc[0]])
108-
109-
span := text[loc[0]:loc[1]]
110-
human := m.ToHumanNameLower(leafIdent(span))
111-
if isSentenceStart(text, loc[0]) {
112-
human = capitalizeFirst(human)
164+
for _, s := range spans {
165+
if s.lo < last {
166+
continue // overlap guard (defensive; spans are non-overlapping)
113167
}
114-
b.WriteString(human)
115-
116-
last = loc[1]
168+
b.WriteString(text[last:s.lo])
169+
b.WriteString(replacement(s, text, o))
170+
last = s.hi
117171
}
118172
b.WriteString(text[last:])
119173

120174
return b.String()
121175
}
122176

123-
// leafIdent extracts the trailing identifier of a doc-link span: it strips the
124-
// surrounding brackets and an optional leading `*`, then keeps the last
125-
// dot-separated segment (`[*inventory.Ledger]` → "Ledger").
126-
func leafIdent(span string) string {
127-
inner := strings.TrimPrefix(span[1:len(span)-1], "*")
128-
if i := strings.LastIndex(inner, "."); i >= 0 {
129-
return inner[i+1:]
177+
// collectSpans gathers the leading self-name (when it matches Self.Name) and
178+
// every doc-link bracket span, in ascending position order.
179+
func collectSpans(text string, o Options) []span {
180+
var spans []span
181+
182+
if o.Self != nil && o.Self.Name != "" {
183+
if loc := identRE.FindStringIndex(strings.TrimLeft(text, " \t")); loc != nil {
184+
offset := len(text) - len(strings.TrimLeft(text, " \t"))
185+
lo, hi := offset+loc[0], offset+loc[1]
186+
if text[lo:hi] == o.Self.Name {
187+
spans = append(spans, span{lo: lo, hi: hi, ref: o.Self.Name, leaf: o.Self.Name, self: true})
188+
}
189+
}
190+
}
191+
192+
for _, loc := range docLinkRE.FindAllStringIndex(text, -1) {
193+
inner := strings.TrimPrefix(text[loc[0]+1:loc[1]-1], "*")
194+
spans = append(spans, span{lo: loc[0], hi: loc[1], ref: inner, leaf: leafOf(inner)})
195+
}
196+
197+
return spans
198+
}
199+
200+
// replacement computes the substitution text for one span: a marker when the
201+
// reference resolves to a schema, otherwise the humanized leaf (sentence-cased
202+
// at the start of the prose). A self-name without a Resolver is left verbatim
203+
// (its exposed name is unknowable without resolution).
204+
func replacement(s span, text string, o Options) string {
205+
titleize := isSentenceStart(text, s.lo)
206+
fallback := o.humanize(s.leaf)
207+
208+
if o.Resolver != nil {
209+
if s.self {
210+
return encodeMarker(o.Self.DefKey, "", fallback, titleize)
211+
}
212+
if res, ok := o.Resolver(s.ref); ok {
213+
return encodeMarker(res.DefKey, res.Suffix, fallback, titleize)
214+
}
215+
} else if s.self {
216+
return text[s.lo:s.hi]
217+
}
218+
219+
if titleize {
220+
return capitalizeFirst(fallback)
221+
}
222+
223+
return fallback
224+
}
225+
226+
// humanize renders an identifier as lower-case words; a nil mangler is treated
227+
// as the identity (defensive — callers always supply one).
228+
func (o Options) humanize(ident string) string {
229+
if o.Mangler == nil {
230+
return ident
231+
}
232+
233+
return o.Mangler.ToHumanNameLower(ident)
234+
}
235+
236+
// leafOf returns the trailing dot-separated segment of a reference chain
237+
// (`inventory.Ledger` → "Ledger").
238+
func leafOf(ref string) string {
239+
if i := strings.LastIndex(ref, "."); i >= 0 {
240+
return ref[i+1:]
130241
}
131242

132-
return inner
243+
return ref
133244
}
134245

135246
// isSentenceStart reports whether pos is the first non-space byte of text — the

‎internal/builders/godoclink/godoclink_test.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ func TestClean(t *testing.T) {
6060

6161
for _, tc := range cases {
6262
t.Run(tc.name, func(t *testing.T) {
63-
assert.Equal(t, tc.want, Clean(tc.in, &m))
63+
assert.Equal(t, tc.want, Clean(tc.in, Options{Mangler: &m}))
6464
})
6565
}
6666
}
@@ -69,5 +69,5 @@ func TestCleanLeavesInitialismCasing(t *testing.T) {
6969
m := mangling.NewNameMangler()
7070
// ToHumanNameLower keeps recognized initialisms cased; CleanGoDoc must not
7171
// re-case them mid-sentence.
72-
assert.Equal(t, "Serves over an HTTP server.", Clean("Serves over an [HTTPServer].", &m))
72+
assert.Equal(t, "Serves over an HTTP server.", Clean("Serves over an [HTTPServer].", Options{Mangler: &m}))
7373
}
Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
// SPDX-FileCopyrightText: Copyright 2015-2025 go-swagger maintainers
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
package godoclink
5+
6+
import (
7+
"regexp"
8+
"strings"
9+
)
10+
11+
// Markers bridge the two halves of idiom recomposition. At the consumption seam
12+
// a resolvable doc-link is encoded as a marker carrying the referenced schema's
13+
// fully-qualified definition key (plus an exposed field-chain suffix, a
14+
// humanized fallback, and a sentence-initial titleize bit). After the spec
15+
// builder reduces definition names, SubstituteMarkers rewrites each marker to
16+
// the schema's final exposed name.
17+
//
18+
// The delimiters are NUL (\x00) and Unit Separator (\x1f): neither can appear in
19+
// a Go source comment, so a marker never collides with real prose. The format is
20+
//
21+
// \x00gl\x1f<defKey>\x1f<suffix>\x1f<fallback>\x1f<0|1>\x00
22+
const (
23+
markerOpen = "\x00gl\x1f"
24+
markerClose = "\x00"
25+
markerSep = "\x1f"
26+
)
27+
28+
// markerRE matches one encoded marker, capturing defKey, suffix, fallback and
29+
// the titleize bit. The field bodies exclude the two delimiter runes so a marker
30+
// can never swallow an adjacent one.
31+
var markerRE = regexp.MustCompile("\x00gl\x1f([^\x1f\x00]*)\x1f([^\x1f\x00]*)\x1f([^\x1f\x00]*)\x1f([01])\x00")
32+
33+
// encodeMarker builds a marker for a resolvable doc-link. defKey is the
34+
// referenced type's definition key; suffix is the exposed field chain (already
35+
// resolved, e.g. ".customer_name") or ""; fallback is the humanized leaf used
36+
// when the key turns out not to be an emitted definition; titleize records
37+
// sentence-initial position.
38+
func encodeMarker(defKey, suffix, fallback string, titleize bool) string {
39+
bit := "0"
40+
if titleize {
41+
bit = "1"
42+
}
43+
44+
return markerOpen + defKey + markerSep + suffix + markerSep + fallback + markerSep + bit + markerClose
45+
}
46+
47+
// HasMarkers reports whether s contains any godoclink marker — a cheap guard so
48+
// callers can skip the substitution walk for marker-free prose.
49+
func HasMarkers(s string) bool {
50+
return strings.Contains(s, markerOpen)
51+
}
52+
53+
// SubstituteMarkers rewrites every marker in text to its final exposed name.
54+
// finalName maps a definition key to the name it is ultimately emitted under,
55+
// returning ok=false when the key is not an emitted definition (pruned or
56+
// unresolved); in that case the marker collapses to its humanized fallback. A
57+
// resolved marker yields finalName+suffix. The sentence-initial bit, when set,
58+
// upper-cases the first rune of the result. No marker ever survives this pass.
59+
func SubstituteMarkers(text string, finalName func(defKey string) (string, bool)) string {
60+
if !HasMarkers(text) {
61+
return text
62+
}
63+
64+
return markerRE.ReplaceAllStringFunc(text, func(m string) string {
65+
groups := markerRE.FindStringSubmatch(m)
66+
defKey, suffix, fallback, bit := groups[1], groups[2], groups[3], groups[4]
67+
68+
var out string
69+
if name, ok := finalName(defKey); ok {
70+
out = name + suffix
71+
} else {
72+
out = fallback
73+
}
74+
if bit == "1" {
75+
out = capitalizeFirst(out)
76+
}
77+
78+
return out
79+
})
80+
}

0 commit comments

Comments
 (0)
⌂ 换网址