A Go HTML scraping library with CSS and XPath selectors, fluent traversal, and schema-based extraction.
go get github.com/your-org/twigpage, err := gosoup.NewPage(rawHTML, "https://example.com")
if err != nil {
log.Fatal(err)
}
title, err := page.CSS("h1").Text()
href, err := page.CSS("a.main-link").Href()Both Page and Builder support CSS and XPath queries. Queries return a *Builder (single match) or Nodes (multiple matches).
// Single match
b := page.CSS("div.article")
b := page.XPath("//div[@class='article']")
// All matches
ns := page.CSSAll("ul li")
ns := page.XPathAll("//ul/li")
// Scoped to a node
b := page.CSS("div.article").CSS("h2")XPath note: Use relative expressions (
.//...) to scope to the current node. Absolute expressions (//...) always search from the document root.
b.Text() // inner text (all descendants)
b.HTML() // outer HTML
b.Attr("data-id") // attribute value
b.Href() // resolved href URL
b.Src() // resolved src URL
b.Tag() // tag name
b.HasClass("foo") // bool
b.NodeType() // html.NodeTypeb.Parent()
b.Child() // first element child
b.Child(2) // nth element child (0-indexed)
b.Next() // next element sibling
b.Prev() // previous element sibling
b.Children() // → Nodes
b.Siblings() // → Nodes
b.Parents() // → Nodes
b.NextAll() // → Nodes
b.PrevAll() // → Nodesns.First() // → *Builder
ns.Last() // → *Builder
ns.Get(i) // → *Builder
ns.Len() // (int, error)
ns.Each(func(i int, b *Builder) { ... })
ns.TextAll() // []string
ns.AttrAll("href") // []string
// Chain further queries
ns.CSS("span")
ns.XPath(".//span")
// Predicates
ns.Some() // any nodes?
ns.None() // no nodes?
ns.Any(func(b *Builder) bool { })
ns.Every(func(b *Builder) bool { })Extract structured data from a node or page in one call. Partial results are always returned even when some fields fail.
schema := twig.Schema{
"title": twig.Field().CSS("h1").Text(),
"link": twig.Field().CSS("a").Href(),
"id": twig.Field().CSS("article").Attr("data-id"),
}
result, err := page.Extract(schema)
// result["title"], result["link"], result["id"]results, err := page.CSSAll("ul li").ExtractAll(schema)result, err := page.Extract(schema)
if err != nil {
if ee, ok := err.(*twig.ExtractionError); ok {
for field, ferr := range ee.Fields {
log.Printf("field %s failed: %v", field, ferr)
}
}
}
// result still contains successfully extracted fields| Error | Meaning |
|---|---|
ErrNotFound |
Selector matched nothing |
ErrInvalidSelector |
Bad CSS or XPath expression |
ErrParseFailed |
HTML parse failure |
ErrIndexOutOfBounds |
Index out of range |
*ExtractionError |
Per-field errors from Extract / ExtractAll |
Errors propagate lazily through builder chains — you only need to check at the terminal call.
// Panics on error — useful in tests and scripts
page := twig.Must(twig.NewPage(rawHTML, ""))
text := twig.Must(page.CSS("h1").Text())