Documentation
¶
Overview ¶
Package web assembles websites from the components of the html package.
A Page is the metadata and content of a single page, independent of where it comes from and how it is rendered: a PageSource yields pages (from markdown files with front matter, a database, …) and a PageRenderer turns one into a complete html.Document.
Site holds what all pages of a website have in common — above all the BaseURL every absolute URL of the site is built from — and derives the site-level files search engines expect from its pages: a Sitemap listing every indexable page and a robots.txt (Robots) pointing at it. It also adds the per-page document metadata that a rendered page needs to be presentable in search results and link previews (canonical URL, description, Open Graph).
Index ¶
- Constants
- Variables
- func AcceptsMarkdown(request *http.Request) bool
- func DefaultRenderPage(ctx context.Context, page *Page) (doc html.Document, err error)
- func MarkdownPath(pagePath string) (string, error)
- type Alternate
- type AlternateRel
- type ChangeFreq
- type ContentRenderer
- type ContentRendererFunc
- type GlobPageSource
- type LLMsLink
- type LLMsSection
- type LLMsTxt
- type LLMsTxtOptions
- type MarkdownDocument
- type MarkdownSection
- type Page
- type PageRenderer
- type PageRendererFunc
- type PageSource
- type PageSourceFunc
- type PageTypeRenderer
- type Robots
- type RobotsGroup
- type Site
- func (s *Site) AddPageMetadata(doc *html.Document, page *Page) error
- func (s *Site) HandleLLMsTxt(response http.ResponseWriter, request *http.Request)
- func (s *Site) HandleRobotsTxt(response http.ResponseWriter, request *http.Request)
- func (s *Site) HandleSitemapXML(response http.ResponseWriter, request *http.Request)
- func (s *Site) LLMsTxt(ctx context.Context, opts LLMsTxtOptions) (*LLMsTxt, error)
- func (s *Site) MarkdownAlternate(page *Page) (*MarkdownDocument, error)
- func (s *Site) MarkdownAlternateURL(page *Page) (string, error)
- func (s *Site) PageHeaders(page *Page) (http.Header, error)
- func (s *Site) PageLang(page *Page) string
- func (s *Site) PageURL(page *Page) (string, error)
- func (s *Site) Pages(ctx context.Context, withContent bool) iter.Seq2[*Page, error]
- func (s *Site) RegisterRoutes(mux *http.ServeMux) (err error)
- func (s *Site) RenderPage(ctx context.Context, page *Page) (html.Document, error)
- func (s *Site) RobotsTxt() (*Robots, error)
- func (s *Site) ServeMarkdown(response http.ResponseWriter, request *http.Request, page *Page, etag string, ...)
- func (s *Site) ServeMarkdownDocument(response http.ResponseWriter, request *http.Request, page *Page, ...)
- func (s *Site) Sitemap(ctx context.Context) (*Sitemap, error)
- func (s *Site) SitemapPathOrDefault() string
- func (s *Site) SitemapRoutePath() (string, error)
- func (s *Site) URL(path string) (string, error)
- func (s *Site) Validate() error
- type Sitemap
- type SitemapIndex
- type SitemapIndexEntry
- type SitemapURL
- type Translation
Constants ¶
const ( // DefaultSitemapPath is the conventional path of a site's sitemap. Unlike // [RobotsTxtPath] the path is not mandated, but it is not free either: a // sitemap only covers URLs at or below its own directory unless it was // submitted through a search console, so a sitemap under "/private/" // cannot list the whole site. Keep it at the root of the site (see // [Site.SitemapPath]). DefaultSitemapPath = "/sitemap.xml" // SitemapNamespace is the XML namespace of the sitemap protocol 0.9, the // current version. SitemapNamespace = "http://www.sitemaps.org/schemas/sitemap/0.9" // MaxSitemapURLs is the maximum number of URLs one sitemap file may // contain. Split a bigger site into several sitemaps and reference them // from a [SitemapIndex]. MaxSitemapURLs = 50000 // MaxSitemapLocLength is the maximum length of a sitemap <loc> URL. MaxSitemapLocLength = 2048 // MaxSitemapBytes is the maximum uncompressed size of one sitemap file. // It is the limit that bites first for a site with long URLs: 50000 URLs // of maximum length would be twice this size. MaxSitemapBytes = 50 << 20 )
const DefaultLLMsSectionTitle = "Pages"
DefaultLLMsSectionTitle is the section title Site.LLMsTxt lists pages without a Page.Type under.
const HrefLangDefault = "x-default"
HrefLangDefault is the hreflang value "x-default" marking the version of a page shown to visitors whose language has no translation of its own.
const LLMsTxtPath = "/llms.txt"
LLMsTxtPath is the path an llms.txt file is served from, at the root of the site like robots.txt.
const RobotsTxtPath = "/robots.txt"
RobotsTxtPath is the only path a robots.txt file can be served from: crawlers request it from the root of the host and ignore a robots.txt served anywhere else (RFC 9309, section 2.3).
Variables ¶
var DefaultMaxPageFileSize int64 = 10 << 20
DefaultMaxPageFileSize is the default of GlobPageSource.MaxFileSize: the size in bytes past which a page file fails its source.
Functions ¶
func AcceptsMarkdown ¶
AcceptsMarkdown reports whether the request's Accept header prefers text/markdown over text/html: it has to name text/markdown explicitly, with a higher quality value than text/html gets, either directly or through a "text/*" or "*/*" wildcard. A missing quality value is 1.
At equal quality the client's order decides: text/markdown listed before text/html, or named while text/html is only reached through a wildcard, wins. RFC 9110 gives the order no meaning, but agents rank equals by it — "text/markdown, text/html, */*" asks for Markdown first.
A browser's Accept header ends in "*/*" and names no Markdown, so it gets HTML; a client that sends "text/markdown" gets Markdown. Wrap the HTML handler of a page with it to serve the alternate under the page's own URL. Both answers then depend on the Accept header, so the HTML response needs "Vary: Accept" as much as the Markdown one, or a cache that kept a browser's HTML hands it to the next agent asking for Markdown. A CDN that ignores Vary for HTML defeats this either way; link agents to the alternate's own URL (see Site.MarkdownAlternateURL) behind one:
response.Header().Add("Vary", "Accept")
if page.HasMarkdownSource() && web.AcceptsMarkdown(request) {
site.ServeMarkdown(response, request, page, etag, lastMod)
return
}
A page without a Markdown source has no alternate, so it answers with HTML whatever the client prefers.
func DefaultRenderPage ¶
DefaultRenderPage renders a Page into an html.Document containing only the document-level metadata derived from the page (title and robots meta), without rendering the page content into the document body.
The robots meta tag marks every page that is not Page.Indexable, which includes drafts and pages scheduled for later, not only pages marked NoIndex — a draft served from a preview deployment must not be indexed either.
func MarkdownPath ¶
MarkdownPath returns the path of the Markdown alternate of the page at pagePath: ".md" appended to the path, or "index.md" for a path ending in a slash, so "/en/faq" becomes "/en/faq.md", "/en/" becomes "/en/index.md" and "/" becomes "/index.md".
It is an error for pagePath to already end in ".md", which would make the page and its alternate the same URL, to end in "/index", whose alternate would be that of the directory page ("/en/index" and "/en/" both mapping to "/en/index.md"), or not to be a valid path within a site.
Types ¶
type Alternate ¶
type Alternate struct {
// Rel is the link relation, [RelAlternate] when empty, or
// [RelDescribedBy].
Rel AlternateRel
// Type is the media type of the alternate, for example "text/markdown".
// It is empty for a translation, which has the type of the page.
Type string
// HrefLang is the BCP 47 language tag of the alternate, or
// [HrefLangDefault]. [Site.DefaultLang] adds the x-default link for most
// sites; set it here only to point it somewhere else, like a language
// chooser.
HrefLang string
// Path is the absolute path of the alternate within the site, validated
// like [Page.Path].
Path string
}
Alternate is another representation of a Page, linked from the page as <link rel="alternate"> in its HTML head and as a Link header of its response, from the same list so the two cannot drift apart.
An alternate with a HrefLang and no Type is a translation: the same page in another language. Search engines use translations to send each visitor to the version in their language, which only works if every version links all the others and itself. The link to itself is added by the Site from the page's language, so Alternates lists only the other versions:
page := &web.Page{
Path: "/en/faq",
Lang: "en",
Alternates: []web.Alternate{
{HrefLang: "de", Path: "/de/faq"},
{Type: "text/markdown", Path: "/en/faq.md"},
},
}
func (Alternate) IsTranslation ¶
IsTranslation reports whether the alternate is a translation of the page: an alternate with a language and the media type of the page itself. An x-default link counts too, because it belongs to the same set of links.
func (Alternate) RelOrDefault ¶
func (a Alternate) RelOrDefault() AlternateRel
RelOrDefault returns Rel, or RelAlternate if it is empty.
func (Alternate) Validate ¶
Validate returns an error if Rel is neither RelAlternate nor RelDescribedBy, Type is not a media type, HrefLang is not a language tag, Path is not a valid path within a site, or if an alternate has neither a Type nor a HrefLang, which would leave nothing to tell it apart from the page.
type AlternateRel ¶
type AlternateRel string //#enum
AlternateRel is the link relation of an Alternate.
const ( // RelAlternate is the link relation of an alternate representation of a // page: a translation, or the same page in another format. RelAlternate AlternateRel = "alternate" // RelDescribedBy is the link relation of a document describing the page, // for example its metadata in another format. RelDescribedBy AlternateRel = "describedby" )
func (AlternateRel) EnumStrings ¶
func (AlternateRel) EnumStrings() []string
EnumStrings returns all valid values for AlternateRel as strings
func (AlternateRel) Enums ¶
func (AlternateRel) Enums() []AlternateRel
Enums returns all valid values for AlternateRel
func (AlternateRel) String ¶
func (a AlternateRel) String() string
String implements the fmt.Stringer interface for AlternateRel
func (AlternateRel) Valid ¶
func (a AlternateRel) Valid() bool
Valid indicates if a is any of the valid values for AlternateRel
func (AlternateRel) ValidOr ¶
func (a AlternateRel) ValidOr(defaultVal AlternateRel) AlternateRel
ValidOr returns a if it is any of the valid values for AlternateRel, else defaultVal
func (AlternateRel) Validate ¶
func (a AlternateRel) Validate() error
Validate returns an error if a is none of the valid values for AlternateRel
type ChangeFreq ¶
type ChangeFreq string //#enum
ChangeFreq tells crawlers how often the content of a page is expected to change. It is a hint that search engines are free to ignore — Google does — so leaving it empty (which omits the <changefreq> element) is a reasonable default.
const ( // ChangeFreqAlways describes a page that changes on every access. ChangeFreqAlways ChangeFreq = "always" // ChangeFreqHourly describes a page that changes hourly. ChangeFreqHourly ChangeFreq = "hourly" // ChangeFreqDaily describes a page that changes daily. ChangeFreqDaily ChangeFreq = "daily" // ChangeFreqWeekly describes a page that changes weekly. ChangeFreqWeekly ChangeFreq = "weekly" // ChangeFreqMonthly describes a page that changes monthly. ChangeFreqMonthly ChangeFreq = "monthly" // ChangeFreqYearly describes a page that changes yearly. ChangeFreqYearly ChangeFreq = "yearly" // ChangeFreqNever describes an archived page that will not change again. ChangeFreqNever ChangeFreq = "never" )
func (ChangeFreq) EnumStrings ¶
func (ChangeFreq) EnumStrings() []string
EnumStrings returns all valid values for ChangeFreq as strings
func (ChangeFreq) Enums ¶
func (ChangeFreq) Enums() []ChangeFreq
Enums returns all valid values for ChangeFreq
func (ChangeFreq) String ¶
func (c ChangeFreq) String() string
String implements the fmt.Stringer interface for ChangeFreq
func (ChangeFreq) Valid ¶
func (c ChangeFreq) Valid() bool
Valid indicates if c is any of the valid values for ChangeFreq
func (ChangeFreq) ValidOr ¶
func (c ChangeFreq) ValidOr(defaultVal ChangeFreq) ChangeFreq
ValidOr returns c if it is any of the valid values for ChangeFreq, else defaultVal
func (ChangeFreq) Validate ¶
func (c ChangeFreq) Validate() error
Validate returns an error if c is none of the valid values for ChangeFreq
type ContentRenderer ¶
type ContentRenderer interface {
RenderContent(ctx context.Context, contentType string, content any) (mx.Component, error)
}
ContentRenderer renders a page's content of the given contentType into a Component.
type ContentRendererFunc ¶
type ContentRendererFunc func(ctx context.Context, contentType string, content any) (mx.Component, error)
ContentRendererFunc is an adapter that lets an ordinary function be used as a ContentRenderer.
func (ContentRendererFunc) RenderContent ¶
func (f ContentRendererFunc) RenderContent(ctx context.Context, contentType string, content any) (mx.Component, error)
RenderContent calls f and implements the ContentRenderer interface.
type GlobPageSource ¶
type GlobPageSource struct {
// Dir is the directory the source reads from. It is both what Pattern is
// matched below and what the URL path of a page is relative to, so the two
// cannot disagree: "content/blog/hello.md" matched by a Dir of
// "content/blog" and a Pattern of "*.md" becomes the page path "/hello".
// Dir is no pattern: a directory named "content[prod]" is matched below
// as it is named, only Pattern holds wildcards.
//
// An empty Dir keeps Pattern as the whole path to match, relative to the
// working directory, and takes the base for page paths from the part of
// Pattern before its first wildcard — the same result written differently,
// down to the error a missing directory fails the source with.
//
// Dir has to name a directory on the local file system; any other fails
// the source. On Windows go-fs reads a file URI with a host, like
// "file://server/share", as the network share `\\server\share`.
//
// Dir is trusted unless Confine is set: the glob and the reads follow
// symlinks, so a symlink below Dir publishes the file it points to,
// wherever that is, as does a ".." in Pattern. Only those who may publish
// any file the process can read should be able to write below Dir.
Dir fs.File
// Confine keeps the source inside Dir, for content that people who may
// not read the server can write, like a contributed content repository.
// The glob and the reads go through an [os.Root] opened on Dir: a symlink
// is followed only if it is relative and stays below Dir, a file or
// directory reached through any other symlink fails the source, and a
// Pattern reaching above Dir with ".." is an error. Confine requires Dir. It is off by default because deploy tools
// symlink content directories, which it would reject.
Confine bool
// Pattern is the glob matched below Dir, in the syntax of filepath.Match.
// It must be relative when Dir is set. Like a shell glob, a wildcard does
// not match a name beginning with ".": a pattern has to name the dot to
// match hidden files, as ".well-known/*.txt" does. Directories, named
// pipes, sockets and devices are skipped, as is a file that is gone by the
// time its turn comes. A page file is opened also when no content is
// read, so one that cannot be read fails the source then too. A directory the glob has to search but cannot read
// fails the source, rather than leaving its pages out without a word; a
// hidden directory is not searched unless the pattern names its dot. Like
// a missing Dir, a missing directory in the fixed part of Pattern, the
// names before the first wildcard, fails the source: "blog/*.md" without
// a "blog" directory is an error, not an empty source. A pattern without
// a wildcard may name a missing file, which matches nothing.
Pattern string
// MaxFileSize is the size in bytes past which a page file fails the
// source instead of being read into memory. 0 means
// [DefaultMaxPageFileSize], a negative value no limit. A sitemap, llms.txt
// or page list, which need no content, read only the head of a Markdown
// file holding its front matter.
MaxFileSize int64
PageType string
DefaultAuthor string
}
GlobPageSource is a PageSource that yields pages from files matching a glob Pattern below Dir. Markdown files are parsed for frontmatter metadata (title, description, lang, date, publishDate, draft, tags), which is not part of the page content and is kept whole, the fields of no page field included, in Page.FrontMatter. HTML and plain-text files are loaded with their respective content types. A lang holding a locale like "en_US" becomes the language tag "en-US", and one that is no language tag fails the source with the name of the file, like front matter that cannot be parsed.
source := &web.GlobPageSource{Dir: fs.File("content/blog"), Pattern: "*.md"}
// content/blog/hello.md -> Page{Path: "/hello"}
func (*GlobPageSource) Pages ¶
Pages iterates over the files matching the glob Pattern and yields a Page for each supported file type, implementing the PageSource interface.
The context is checked while the glob reads each directory and between the chunks of every file read, so a cancelled request stops at the next of them. A single call into the operating system that stalls, a stat, an open or a read on a hung network or FUSE mount, cannot be interrupted: the source returns with the context's error once that call does.
type LLMsLink ¶
type LLMsLink struct {
// Title is the text of the link.
Title string
// URL is an absolute http or https URL or a path starting with "/".
URL string
// Note is an optional description of what is behind the link.
Note string
}
LLMsLink is one link of an LLMsSection, rendered as "- [Title](URL): Note".
type LLMsSection ¶
type LLMsSection struct {
// Title is the heading of the section.
Title string
// Links are rendered as a list in the given order.
Links []LLMsLink
}
LLMsSection is one section of an LLMsTxt: a heading and a list of links.
func (LLMsSection) Validate ¶
func (s LLMsSection) Validate() error
Validate returns an error if the section title is empty or not a single line, or if one of its links is invalid.
type LLMsTxt ¶
type LLMsTxt struct {
// Title is the name of the site or project, the one required field.
Title string
// Summary is a short summary of the site, rendered as a block quote.
Summary string
// Context is a paragraph of further information on how to read the site,
// rendered after the summary.
Context string
// Sections list the links, each under a "## Title" heading.
Sections []LLMsSection
}
LLMsTxt is an llms.txt file (https://llmstxt.org): a Markdown index of a site written for language models, which have little use for navigation, scripts and layout and a limited context to read a site with. It names the site, summarizes it and links the pages worth reading in sections, ideally their Markdown alternates (see Site.MarkdownAlternate):
# Title > Summary Context ## Section - [Link title](https://example.com/page.md): Note
A section titled "Optional" has a meaning of its own: its links may be skipped by a reader that needs a shorter context.
Like Robots, it validates before writing, and for the same reason: the format is line based, so a line break smuggled into a value would add a heading or a link nobody wrote. Site.LLMsTxt builds one from the pages of a site.
func (*LLMsTxt) HandleHTTP ¶
func (l *LLMsTxt) HandleHTTP(response http.ResponseWriter, request *http.Request)
HandleHTTP writes the llms.txt file as a text/markdown response, implementing http.HandlerFunc. The ETag is a hash of the content, so a client re-reading an unchanged file gets a 304. On a validation error it responds with a generic 500 status via mx.RespondNonContextError.
func (*LLMsTxt) MarshalText ¶
MarshalText renders the llms.txt file, implementing encoding.TextMarshaler. It returns the error from LLMsTxt.Validate instead of writing a file whose structure would differ from what the value describes.
func (*LLMsTxt) String ¶
String renders the llms.txt file, returning a "!ERROR: " line instead of the file content if the value does not validate. It is meant for debugging and tests; use LLMsTxt.MarshalText to handle the error.
func (*LLMsTxt) Validate ¶
Validate returns an error if the Title is missing, or if a field holds what would change the structure of the rendered file: a line break or another control character in any field, a context that would open a block of its own (a heading, a block quote, a code block or fence, raw HTML), an empty section or link title, a bracket, backtick, angle bracket or trailing backslash in a link title, which would end, escape or swallow the link text, a link title that is a lone "x", which GFM renders as a checked task list box instead of the link, a parenthesis, angle bracket, backslash or whitespace in a URL, which would end or escape the link destination, or a URL that is neither an absolute http or https URL nor a path within the site.
type LLMsTxtOptions ¶
type LLMsTxtOptions struct {
// Title of the llms.txt, Site.Title when empty.
Title string
// Summary of the llms.txt, Site.Description when empty.
Summary string
// Context is the optional paragraph after the summary.
Context string
// SectionTitles maps a Page.Type to the title of the section listing the
// pages of that type. A type without an entry is used as the title
// itself, and pages without a type are listed under
// [DefaultLLMsSectionTitle] unless the map has an entry for "".
SectionTitles map[string]string
}
LLMsTxtOptions configure the llms.txt built by Site.LLMsTxt.
type MarkdownDocument ¶
type MarkdownDocument struct {
// Title is rendered as the H1 heading, Description as a block quote.
// Empty values are left out.
Title string
Description string
// URL is the canonical URL of the page, the HTML version the document is
// an alternate of.
URL string
// Lang is the BCP 47 language tag of the page.
Lang string
// LastUpdated is the time the page content last changed, rendered as an
// RFC 3339 date. The zero time is left out.
LastUpdated time.Time
// Translations are the versions of the page in other languages. One
// whose Lang leaves no usable link text is left out.
Translations []Translation
// Sections are rendered after the metadata and before the page's
// Markdown source.
Sections []MarkdownSection
// Source is the Markdown source of the page, rendered verbatim after a
// horizontal rule.
Source []byte
}
MarkdownDocument is the Markdown alternate of a page: the page's own Markdown source, preceded by a header that makes the document make sense on its own — for a reader that fetched it without the HTML page around it, like a crawler or an AI agent:
# Title > Description - Canonical URL: https://example.com/en/faq - Language: en - Last updated: 2026-08-20 - Translations: [de](https://example.com/de/faq) ## Section title Section body --- The page's Markdown source, verbatim.
Site.MarkdownAlternate builds one from a page. Its fields are rendered only when the document is written, so a caller can still add Sections — the facts about a page that only the caller knows — after building it.
func (*MarkdownDocument) Bytes ¶
func (d *MarkdownDocument) Bytes() []byte
Bytes renders the document.
type MarkdownSection ¶
type MarkdownSection struct {
// Title is the heading of the section. Line breaks are rendered as
// spaces, because a heading ends at the end of its line. An empty Title
// renders the Body without a heading.
Title string
// Body is the Markdown content of the section, rendered verbatim.
Body string
}
MarkdownSection is a section a caller adds to a MarkdownDocument, rendered as a "## Title" heading followed by the Body Markdown.
type Page ¶
type Page struct {
// Route mx.Route
PathValues map[string]any
// Path is the absolute URL path of the page within the site, like
// "/blog/hello-world". A [Site] needs it to build the page's canonical
// URL: without it the page renders without a canonical link, and an
// [Page.Indexable] page without it fails the sitemap it should be part of.
Path string
// Lang is the BCP 47 language tag of the page, for example "de". It falls
// back to Site.Lang when empty. A site in several languages needs it for
// every page with translations, because the hreflang links of a page have
// to name the page's own language too (see [Alternate]).
Lang string
Title string
// Description is the summary shown by search engines and link previews.
// It falls back to Site.Description when empty.
Description string
Author string
Type string
Tags []string
NoIndex bool // <meta name="robots" content="noindex, nofollow" />
Created time.Time
LastUpdated time.Time
Published time.Time // Zero time means not published
// Alternates are the other representations of the page: translations,
// other formats like a Markdown version, or a document describing it.
// [Site.AddPageMetadata] renders them as <link> elements and
// [Site.PageHeaders] as Link headers.
Alternates []Alternate
// FrontMatter holds every field of the front matter the page's source was
// written with, keyed by name as written, including those a [PageSource]
// maps to the fields above, so a [PageRenderer] can read fields of its
// own, like an "image" or a "layout". Nested objects are map[string]any
// whatever the format, values otherwise as the YAML, TOML or JSON decoder
// of the format made them. It is nil for a page without front matter;
// [GlobPageSource] fills it for Markdown files, also when no content is
// read. No surface publishes it: the Markdown alternate of a page carries
// a header built from the fields above and the Content, which holds no
// front matter.
FrontMatter map[string]any
Resources []fs.File // URLs or file paths
ContentType string
Content any // Can be nil if only metadata is needed
}
Page holds the metadata and content of a single web page, independent of how it is sourced or rendered.
func (*Page) HasMarkdownSource ¶
HasMarkdownSource reports whether the page has a Markdown alternate, like the ok result of Page.MarkdownSource, without copying string content.
func (*Page) Indexable ¶
Indexable reports whether the page should be offered to search engines: it has to be published and must not be marked NoIndex. It decides both whether Site lists the page in its sitemap and whether it renders a "noindex" robots meta tag for the page.
It is the one predicate for "public": every surface that publishes pages beyond the page itself — the sitemap, llms.txt, the MCP page tools — lists only pages it is true for.
func (*Page) IsPublished ¶
IsPublished reports whether the page is published at the current time, which is the case if Published is neither zero (a draft) nor in the future (a scheduled page).
func (*Page) LastModified ¶
LastModified returns when the content of the page last changed: LastUpdated, or Published if LastUpdated is zero.
func (*Page) MarkdownSource ¶
MarkdownSource returns the Markdown source of the page and true if the page has a Markdown alternate, which is the case if its ContentType is text/markdown (with any parameters, like the charset of mx.ContentTypeMarkdown) and its Content a string or []byte.
Content is only loaded by a PageSource asked for it, so a page iterated without content never has a Markdown alternate.
String content is copied into the returned slice on every call; a caller that only needs to know whether there is a Markdown source asks Page.HasMarkdownSource.
func (*Page) Validate ¶
Validate returns an error if the page's Path is set but not a valid path within a site, if Lang is set but not a language tag, or if one of its Alternates is invalid: malformed, a translation pointing back at the page itself, repeating the rel, type and hreflang of another alternate, which would leave crawlers two answers for one question, or a text/markdown alternate without a language anywhere but at the page's MarkdownPath, the Markdown version llms.txt and the MCP tools link. Another format of the page may share its URL, served by content negotiation (see AcceptsMarkdown).
A page with translations additionally needs a language, which may come from the site; Site.PageHeaders and Site.AddPageMetadata check that too.
type PageRenderer ¶
PageRenderer renders a Page into a complete html.Document.
var DefaultPageRenderer PageRenderer = PageRendererFunc(DefaultRenderPage)
DefaultPageRenderer is the PageRenderer used when no other renderer is configured. It wraps DefaultRenderPage.
type PageRendererFunc ¶
PageRendererFunc is an adapter that lets an ordinary function be used as a PageRenderer.
func (PageRendererFunc) RenderPage ¶
RenderPage calls f and implements the PageRenderer interface.
type PageSource ¶
PageSource provides an iterator over the pages it contains. If withContent is false, only page metadata is loaded and Page.Content is left empty.
type PageSourceFunc ¶
PageSourceFunc is an adapter that lets an ordinary function be used as a PageSource.
type PageTypeRenderer ¶
type PageTypeRenderer struct {
DefaultRenderer PageRenderer
PageTypeRenderers map[string]PageRenderer
ContentTypeRenderers map[string]ContentRenderer
}
PageTypeRenderer is a PageRenderer that dispatches to a per-Page.Type PageRenderer and a per-Page.ContentType ContentRenderer. When the selected page renderer produces a document without a body, the matching content renderer is used to render Page.Content into the body.
func (*PageTypeRenderer) RenderPage ¶
RenderPage selects a PageRenderer by Page.Type (falling back to DefaultRenderer or DefaultPageRenderer), then, if the resulting document has no BodyChildren (nil or empty) and the page has content, renders Page.Content via the ContentRenderer registered for Page.ContentType. It implements the PageRenderer interface.
type Robots ¶
type Robots struct {
// Groups are the crawl rule groups, rendered in order and separated by
// blank lines. No groups at all means an empty robots.txt, which allows
// crawling everything.
Groups []RobotsGroup
// Sitemaps are absolute sitemap URLs rendered as "Sitemap:" lines after
// the groups. They are independent of the groups: every crawler reads all
// of them, which is why a sitemap must be referenced by its full URL.
Sitemaps []string
}
Robots is the content of a robots.txt file: groups of crawl rules addressed at user agents, followed by the URLs of the site's sitemaps.
Robots.MarshalText and Robots.WriteTo validate before writing, so a value that would silently change the meaning of the file — an unanchored path, a line break smuggled into a user agent — fails instead of being rendered.
Use AllowAllRobots and DisallowAllRobots for the two common cases; a Site serves the robots.txt of its Robots field with the URL of its sitemap added (see Site.RobotsTxt).
func AllowAllRobots ¶
AllowAllRobots returns a Robots that allows every crawler to crawl the whole site, optionally referencing the given absolute sitemap URLs.
func DisallowAllRobots ¶
func DisallowAllRobots() *Robots
DisallowAllRobots returns a Robots that asks every crawler to stay away from the whole site, the usual setting for a staging or preview deployment.
It keeps a site out of crawls, not out of search results: a page linked from somewhere else can still be indexed without ever being fetched. Only a "noindex" robots meta tag on a crawlable page (see Page.NoIndex) reliably keeps it out of the index.
func (*Robots) HandleHTTP ¶
func (r *Robots) HandleHTTP(response http.ResponseWriter, request *http.Request)
HandleHTTP writes the robots.txt file as a plain text response, implementing http.HandlerFunc. On a validation error it responds with a generic 500 status via mx.RespondNonContextError.
func (*Robots) MarshalText ¶
MarshalText renders the robots.txt file, implementing encoding.TextMarshaler. It returns the error from Robots.Validate instead of writing a file whose meaning would differ from what the value describes.
func (*Robots) String ¶
String renders the robots.txt file, returning a "!ERROR: " line instead of the file content if the value does not validate. It is meant for debugging and tests; use Robots.MarshalText to handle the error.
type RobotsGroup ¶
type RobotsGroup struct {
// UserAgents are the product tokens of the crawlers this group addresses,
// for example "Googlebot" or "*" for all of them. Matching is
// case-insensitive. No user agents at all renders as "User-agent: *".
UserAgents []string
// Allow are the paths that may be crawled. Because everything that is not
// disallowed may be crawled anyway, an Allow rule only has an effect as
// the exception to a Disallow rule of the same group. Each path must start
// with "/" and may use the wildcards "*" (any sequence of characters) and
// "$" (end of URL).
Allow []string
// Disallow are the paths that must not be crawled, with the same syntax as
// Allow. The empty path is the idiomatic "Disallow:" line that disallows
// nothing.
Disallow []string
// CrawlDelay is the minimum time a crawler should wait between two
// requests. It is not part of RFC 9309 and is ignored by Googlebot, but
// honored by Bing and Yandex. Zero omits the line.
CrawlDelay time.Duration
}
RobotsGroup is one group of crawl rules in a robots.txt file: the user agents it addresses, followed by the rules that apply to them.
A crawler obeys only the one group whose user agent matches it most specifically. Groups do not inherit from each other, so rules for a named bot have to repeat everything from the "*" group that should still apply to it.
func (RobotsGroup) Validate ¶
func (g RobotsGroup) Validate() error
Validate returns an error if a user agent is not a valid product token, a path is not anchored at "/", or a value contains characters that would break the line based robots.txt format.
type Site ¶
type Site struct {
// BaseURL is the absolute URL the site is served from, for example
// "https://example.com" or "https://example.com/blog" for a site below a
// path prefix. It must have an http or https scheme and a host and must not
// have a query or fragment. A trailing slash is optional.
//
// Every absolute URL of the site is built from it, so it has to be the one
// canonical origin of the site: the redirect target of all other spellings
// (www vs. no www, http vs. https), not just whatever host is being served.
BaseURL string
// Title is the name of the whole site, used as the Open Graph og:site_name.
Title string
// Description is the fallback for the description of a page that has none.
Description string
// Lang is the BCP 47 language tag of the site, for example "en" or
// "de-AT". It is the language of every page without a Page.Lang of its
// own, used for the html lang attribute of pages whose document does not
// set one.
Lang string
// DefaultLang is the language of the version of a page shown to visitors
// whose language has no translation of its own. When it is set and a page
// has a translation in this language (or is in this language itself), the
// page's hreflang links get an "x-default" link pointing at that version.
// Empty means no x-default link.
DefaultLang string
// Author is the fallback for the author of a page that has none.
Author string
// Sources are the pages of the site, in the order they are iterated by
// [Site.Pages]. They are what [Site.Sitemap] is built from.
Sources []PageSource
// Renderer renders a single page into an html.Document.
// [DefaultPageRenderer] is used when it is nil.
Renderer PageRenderer
// Robots is the robots.txt served for the site. A nil Robots means an
// allow-all robots.txt. [Site.RobotsTxt] adds the URL of the site's
// sitemap to whatever is configured here.
Robots *Robots
// SitemapPath is the path the sitemap is served from,
// [DefaultSitemapPath] when empty.
SitemapPath string
// Routes are the site's own routes, registered on the mux passed to
// [Site.RegisterRoutes] together with the robots.txt and sitemap routes.
Routes []mx.Route
// LLMsTxtOptions, if set, has [Site.RegisterRoutes] serve the llms.txt
// that [Site.LLMsTxt] builds with these options (see
// [Site.HandleLLMsTxt]) at [LLMsTxtPath] below the BaseURL. Nil serves
// none: pointing agents at the site's pages is a publishing decision of
// its own, not a side effect of having a sitemap.
LLMsTxtOptions *LLMsTxtOptions
// MarkdownAlternates, if set, has [Site.RegisterRoutes] serve the
// Markdown alternate of every public page with a Markdown source at its
// [MarkdownPath], the URL [Site.MarkdownAlternateURL], llms.txt and the
// MCP page tools link. The alternate is served by
// [Site.ServeMarkdownDocument] with a hash of the document as its ETag.
// A site that serves the alternates itself, under the page URL by
// [AcceptsMarkdown], with its own revision as ETag or with Sections of
// its own, leaves it unset.
//
// Every page with a text/markdown ContentType when the Sources are read
// without content gets a route of its own when RegisterRoutes is called,
// because http.ServeMux cannot match a path by its suffix and a catch-all
// pattern would conflict with the routes serving the pages, or take
// their requests. Whether the page is public and what it says is decided
// for every request, so a page published or changed later is served as
// it is then, but the alternate of a page added to the Sources later has
// no route until the routes are registered on a new mux.
MarkdownAlternates bool
}
Site is the configuration a website has in common across all of its pages.
It does not serve the pages themselves — that is what Routes and a PageRenderer are for — but it owns everything that is only knowable at the level of the whole site: the absolute URL of every page, which pages search engines should see, and the robots.txt and sitemap files derived from them.
site := &web.Site{
BaseURL: "https://example.com",
Title: "Example",
Lang: "en",
Sources: []web.PageSource{postsSource},
}
err := site.RegisterRoutes(mux)
func (*Site) AddPageMetadata ¶
AddPageMetadata adds the document metadata that makes a rendered page presentable in search results and link previews. It is called by Site.RenderPage and can be called directly by a custom PageRenderer.
Everything the renderer already set is kept, so a page can always override the site-level default:
- the html lang attribute from Page.Lang or Site.Lang and the <title> from Page.Title
- description and author meta tags, falling back to the site values
- the Open Graph tags describing the page, including the article dates for a published page
- a canonical link and og:url, followed by a link for every Alternate of the page (see Site.PageHeaders for the same links as headers) and an og:locale:alternate tag for every translation, which link previews offer instead of the hreflang links. They are skipped for a page without a Path because there is no URL to point at, and for a document that already carries an og:url, which is taken as the renderer having declared the page URL and its links itself. That og:url has to be the canonical URL of the page, though, because Site.PageHeaders sends that one as a Link header: a document declaring another URL is an error
The one exception is the robots meta tag: a page that is not Page.Indexable is always marked "noindex, nofollow", so that drafts, scheduled pages and pages marked NoIndex cannot end up in search results because a renderer set a robots tag of its own.
func (*Site) HandleLLMsTxt ¶
func (s *Site) HandleLLMsTxt(response http.ResponseWriter, request *http.Request)
HandleLLMsTxt builds the site's llms.txt with Site.LLMsTxt from LLMsTxtOptions, the zero options when it is nil, and writes it as a text/markdown response (see LLMsTxt.HandleHTTP), implementing http.HandlerFunc. Like the sitemap it is built from the Sources for every request, with their content; a site with expensive sources should cache it.
func (*Site) HandleRobotsTxt ¶
func (s *Site) HandleRobotsTxt(response http.ResponseWriter, request *http.Request)
HandleRobotsTxt writes the site's robots.txt as a plain text response, implementing http.HandlerFunc.
func (*Site) HandleSitemapXML ¶
func (s *Site) HandleSitemapXML(response http.ResponseWriter, request *http.Request)
HandleSitemapXML builds the site's sitemap and writes it as an XML response, implementing http.HandlerFunc. The sitemap is built from the Sources for every request; a site with expensive sources should cache it.
func (*Site) LLMsTxt ¶
LLMsTxt builds an llms.txt listing the Markdown alternates of the site's pages: every Page.Indexable page with a Markdown source (see Page.MarkdownSource), linked by the URL of its alternate (see MarkdownPath), with its title and description. The pages are grouped into one section per Page.Type, in the order the types first appear in the Sources, and listed in source order within a section. Types that SectionTitles maps to the same title share one section.
It reads the Sources with content, because only the content tells whether a page has a Markdown source. A site with expensive sources should cache the result.
Page titles and descriptions are site content rather than the structure of the file, so instead of failing the whole file they are made to fit: line breaks become spaces, other control characters are dropped, square brackets in a title become parentheses, backticks apostrophes and "<" a "‹", a trailing backslash is dropped, and the parentheses, spaces and backslashes of a URL are percent-encoded. A page without a usable title — or with one that is a lone "x" (see LLMsLink.Validate) — is listed by its path.
A public page that has no Path, or whose Markdown alternate could not be built, fails the whole file like it fails the sitemap: it is a bug in the source, not something to discover as a silently incomplete index.
It is the easy default. A site that wants other sections, notes or links builds its own LLMsTxt.
func (*Site) MarkdownAlternate ¶
func (s *Site) MarkdownAlternate(page *Page) (*MarkdownDocument, error)
MarkdownAlternate returns the Markdown alternate of the page: its Markdown source (see Page.MarkdownSource) with a header carrying its title, description, canonical URL, language, last update — LastUpdated, falling back to Published — and the URLs of its translations. The source is published as it is, HTML comments included, which an HTML renderer may drop from the page.
It is an error for the page to have no Markdown source or no Path, to fail Page.Validate, or to fail the checks of its links that need the site: a BaseURL, and for a page with translations a valid language of its own (from Page.Lang or Site.Lang) that none of them repeats. Whether the page should be published at all is the caller's decision: unlike the sitemap, this does not check Page.Indexable, so a handler serving the alternates of a site has to, just as it does for the HTML pages.
func (*Site) MarkdownAlternateURL ¶
MarkdownAlternateURL returns the absolute URL of the page's Markdown alternate, at the MarkdownPath of its Path. It fails for a page without a valid Path and for one whose links cannot be built, as Site.MarkdownAlternate does, so the alternate can be served wherever this URL is listed. Whether the page has a Markdown source is not checked.
func (*Site) PageHeaders ¶
PageHeaders returns the Link headers of the page: its canonical URL and its [Alternate]s, built from the same list Site.AddPageMetadata renders into the HTML head, so a crawler reading either finds the same links. A page without a Path has none and gets an empty header.
Every translation is one header line of about a hundred bytes, which adds up for a page in dozens of languages: proxies limit the size of response headers (nginx to 4 KiB by default). The links in the HTML head alone are enough for an HTML page; the headers matter most for responses without a head of their own.
The translations are published as the page lists them. Alternates should name only public pages, since search engines discard a set that points at a page marked noindex, and a draft's URL would be published with it.
Add them to the response of the page before writing it:
header, err := site.PageHeaders(page)
if err != nil { ... }
for _, link := range header.Values("Link") {
response.Header().Add("Link", link)
}
func (*Site) PageLang ¶
PageLang returns the language of the page: its own Lang, or the Lang of the site. It is the language of the html lang attribute, of the page's own hreflang link and of its Markdown alternate, so anything else describing the page should report it too.
func (*Site) PageURL ¶
PageURL returns the absolute canonical URL of the page. It is an error for the page to have no Path, because there is no URL to build then.
func (*Site) Pages ¶
Pages iterates the pages of all Sources in order, implementing the PageSource interface.
func (*Site) RegisterRoutes ¶
RegisterRoutes registers the site's robots.txt and sitemap handlers together with the site's Routes on mux, returning the error from Site.Validate if the site is not configured completely enough to build absolute URLs. With LLMsTxtOptions set it registers Site.HandleLLMsTxt too, and with MarkdownAlternates the Markdown alternates of the pages, for which it reads the Sources once without content and returns their error.
The handlers are registered at their path below the root of the host, so mux has to be the one serving that root. That is not a limitation of this package but of robots.txt: crawlers only ever read it from "/robots.txt", so a site below a path prefix still has to serve it from the root of the host. The sitemap, llms.txt and the alternates are served below the path of the BaseURL, where their URLs point.
func (*Site) RenderPage ¶
RenderPage renders the page with the site's Renderer (DefaultPageRenderer when nil) and adds the site-level document metadata with Site.AddPageMetadata. It implements the PageRenderer interface, so a Site can be used wherever a single-page renderer is expected.
func (*Site) RobotsTxt ¶
RobotsTxt returns the robots.txt of the site: the Robots value with the absolute URL of the site's sitemap added, or an allow-all robots.txt referencing the sitemap when Robots is nil. The Robots field is not modified, so repeated calls return the same file.
func (*Site) ServeMarkdown ¶
func (s *Site) ServeMarkdown(response http.ResponseWriter, request *http.Request, page *Page, etag string, lastMod time.Time)
ServeMarkdown writes the Markdown alternate of the page (see Site.MarkdownAlternate) as a text/markdown response, with the headers and the conditional-request handling of Site.ServeMarkdownDocument. It serves a page as it is; to add Sections to the alternate, or change its other fields, build it with Site.MarkdownAlternate and serve the result with Site.ServeMarkdownDocument instead.
A page without a Markdown source has no alternate to serve and gets a 404; any other failure of Site.MarkdownAlternate is a 500.
func (*Site) ServeMarkdownDocument ¶
func (s *Site) ServeMarkdownDocument(response http.ResponseWriter, request *http.Request, page *Page, doc *MarkdownDocument, etag string, lastMod time.Time)
ServeMarkdownDocument writes doc, the Markdown alternate of the page, as a text/markdown response. It is Site.ServeMarkdown for a document the caller built with Site.MarkdownAlternate and then changed, typically by adding Sections; a page served as it is goes through Site.ServeMarkdown.
The response carries a canonical Link header pointing at doc.URL, the HTML page, so search engines index that one rather than the alternate (none for an empty doc.URL, which the body leaves out too), and "Vary: Accept", because a site serving the alternate under the page's own URL to a client that asks for Markdown (see AcceptsMarkdown) has to keep caches from handing it to everyone else. A page that is not Page.Indexable is marked "X-Robots-Tag: noindex, nofollow", the header form of its robots meta tag. That is all the page is used for: everything else comes from doc.
The etag and lastMod are the caller's revision of the document, or empty and zero for none; for a document with Sections of its own they have to change when those do. The etag is quoted if it is not yet and marked "-md" inside the quotes: under content negotiation the alternate shares its URL with the HTML page, and a cache revalidating both with one tag could answer a Markdown client with the HTML. Conditional and range requests are answered by http.ServeContent, so a client presenting the current tag gets a 304. Headers set before the call, like Cache-Control, are left as they are, and the request is used as given, so a caller that wants only the etag to answer a conditional request passes a clone without If-Modified-Since.
A nil doc is a bug in the caller and gets a 500, like a failure of Site.MarkdownAlternate in Site.ServeMarkdown.
func (*Site) Sitemap ¶
Sitemap builds the sitemap of the site from the Page.Indexable pages of all Sources, sorted by URL so that a regenerated sitemap only differs where the site did. Pages that are drafts, scheduled for later or marked NoIndex are left out, because a sitemap should only list URLs that are meant to show up in search results.
An indexable page without a Path is an error rather than a skipped entry: it cannot be pointed at, which is a bug in the PageSource and not something to discover as a silently incomplete sitemap. So is a page whose links cannot be built (see Page.Validate), because Site.RenderPage fails for it and the sitemap would point crawlers at an error.
func (*Site) SitemapPathOrDefault ¶
SitemapPathOrDefault returns SitemapPath, or DefaultSitemapPath if it is empty. It is the path below the BaseURL, not necessarily the path below the root of the host — see Site.SitemapRoutePath.
func (*Site) SitemapRoutePath ¶
SitemapRoutePath returns the path the sitemap is served from as seen from the root of the host: SitemapPath below the path prefix of the BaseURL. A site with a BaseURL of "https://example.com/blog" and the default sitemap path serves it at "/blog/sitemap.xml".
Site.RegisterRoutes registers this path and Site.RobotsTxt advertises the URL built from it, so what the robots.txt points at is always what the site serves.
func (*Site) URL ¶
URL returns the absolute URL of an absolute path within the site, joining it to the BaseURL. Unescaped characters are escaped, so the result is always a usable URL. A trailing slash of the path is preserved because "/blog" and "/blog/" are different URLs.
See [validateSitePath] for what a path has to look like.
type Sitemap ¶
type Sitemap struct {
// URLs are the sitemap entries, rendered in the given order. There may be
// at most [MaxSitemapURLs] of them.
URLs []SitemapURL
}
Sitemap is a sitemap XML file listing the URLs of a site that should be crawled. It is a mx.Component rendering the complete XML document, so it can be written to a file with xml.String for a statically generated site or served with Sitemap.HandleHTTP.
A sitemap does not decide what gets indexed, it only helps crawlers discover pages they might otherwise miss, and tells them via <lastmod> what is worth re-fetching. Build one from the pages of a Site with Site.Sitemap.
func (*Sitemap) Document ¶
Document returns the sitemap as an XML document. An invalid sitemap yields a document whose root defers the error from Sitemap.Validate to render time (see mx.NewErrElement), so an invalid value can never render as markup. Such a document carries no XML declaration either, so rendering it writes nothing at all before failing.
func (*Sitemap) HandleHTTP ¶
func (s *Sitemap) HandleHTTP(response http.ResponseWriter, request *http.Request)
HandleHTTP renders the sitemap and writes it as an indented XML response, implementing http.HandlerFunc.
func (*Sitemap) Render ¶
Render writes the sitemap XML document, implementing mx.Component. It validates before writing anything, so a render straight into a file or a socket cannot leave a truncated document behind.
func (*Sitemap) Validate ¶
Validate returns an error if the sitemap holds more than MaxSitemapURLs entries or if any entry is invalid.
type SitemapIndex ¶
type SitemapIndex struct {
// Sitemaps are the index entries, rendered in the given order. There may
// be at most [MaxSitemapURLs] of them.
Sitemaps []SitemapIndexEntry
}
SitemapIndex is a sitemap index file listing the URLs of several sitemap files. It is needed for sites with more than MaxSitemapURLs pages and useful to split a site into separately updated sitemaps, for example one per section. Like Sitemap it is a mx.Component rendering the complete XML document.
func (*SitemapIndex) Document ¶
func (x *SitemapIndex) Document() *xml.Document
Document returns the sitemap index as an XML document, deferring a validation error to render time like Sitemap.Document.
func (*SitemapIndex) HandleHTTP ¶
func (x *SitemapIndex) HandleHTTP(response http.ResponseWriter, request *http.Request)
HandleHTTP renders the sitemap index and writes it as an indented XML response, implementing http.HandlerFunc.
func (*SitemapIndex) Render ¶
Render writes the sitemap index XML document, implementing mx.Component. Like Sitemap.Render it validates before writing anything.
func (*SitemapIndex) Validate ¶
func (x *SitemapIndex) Validate() error
Validate returns an error if the index holds more than MaxSitemapURLs entries or if any entry is invalid.
type SitemapIndexEntry ¶
type SitemapIndexEntry struct {
// Loc is the absolute URL of the sitemap file.
Loc string
// LastMod is the time that sitemap file was last modified, rendered as an
// RFC 3339 timestamp. The zero time omits the element.
LastMod time.Time
}
SitemapIndexEntry is one <sitemap> entry of a SitemapIndex.
func (SitemapIndexEntry) Validate ¶
func (e SitemapIndexEntry) Validate() error
Validate returns an error if Loc is not an absolute http or https URL.
type SitemapURL ¶
type SitemapURL struct {
// Loc is the absolute URL of the page, including the scheme and host.
// It is the only required field.
Loc string
// LastMod is the time the page was last modified, rendered as an RFC 3339
// timestamp. The zero time omits the element. It is the one entry field
// search engines actually use, so it should only be set when it reflects a
// real content change — a timestamp that moves on every deploy teaches
// crawlers to ignore it.
LastMod time.Time
// ChangeFreq is an optional hint about how often the page changes.
// The empty value omits the element.
ChangeFreq ChangeFreq
// Priority is the optional priority of this page relative to the other
// pages of the same site, from 0.1 (lowest) to 1.0 (highest). Zero omits
// the element, which means the protocol default of 0.5. It has no effect
// on the ranking between sites and is ignored by Google.
Priority float64
}
SitemapURL is one <url> entry of a Sitemap.
func (SitemapURL) Validate ¶
func (u SitemapURL) Validate() error
Validate returns an error if Loc is not an absolute http or https URL, or if ChangeFreq or Priority are outside their allowed value ranges.
type Translation ¶
type Translation struct {
// Lang is the BCP 47 language tag of the translation.
Lang string
// URL is the absolute URL of the translation.
URL string
}
Translation is a version of a page in another language.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package schemaorg is a typed subset of the schema.org vocabulary for publishing structured data as JSON-LD: the people, organizations, pages, events, offers and FAQs that search engines read from a <script type="application/ld+json"> to understand what a page is about.
|
Package schemaorg is a typed subset of the schema.org vocabulary for publishing structured data as JSON-LD: the people, organizations, pages, events, offers and FAQs that search engines read from a <script type="application/ld+json"> to understand what a page is about. |