Type-safe HTML scraping with XPath selectors and external configuration.
- Type-safe with Go generics
- External config (JSON/YAML)
- XPath validation before scraping
- Pagination support - Auto-follow next-link or numbered pagination
- Fallback XPath chains (
altXpath,altContainer) for handling varying HTML structures - Data transformation pipes (trim, int/float conversion, regex, URL parsing, etc.)
- Custom pipe registration for domain-specific transformations
- Structured logging with configurable levels (slog-based)
- SSRF protection - Blocks private IPs by default
- Health checks for URLs
- Production ready (retries, proxy, timeouts)
go get github.com/Hanivan/gtmlpselectors.json:
{
"container": "//div[@class='product']",
"fields": {
"name": {"xpath": ".//h2/text()"},
"price": {"xpath": ".//span[@class='price']/text()", "pipes": ["trim", "tofloat"]}
}
}main.go:
type Product struct {
Name string `json:"name"`
Price float64 `json:"price"`
}
config, _ := gtmlp.LoadConfig("selectors.json", nil)
products, _ := gtmlp.ScrapeURL[Product](context.Background(), "https://example.com", config)
for _, p := range products {
fmt.Printf("%s: %.2f\n", p.Name, p.Price)
}Or embed config with go:embed:
//go:embed selectors.yaml
var configYAML string
config, _ := gtmlp.ParseConfig(configYAML, gtmlp.FormatYAML, nil)
products, _ := gtmlp.ScrapeURL[Product](context.Background(), "https://example.com", config)Configure log levels for different environments:
import "log/slog"
// Development: see HTTP requests and scraping details
gtmlp.SetLogLevel(slog.LevelInfo)
// Troubleshooting: see XPath evaluation and fallbacks
gtmlp.SetLogLevel(slog.LevelDebug)
// Production: warnings and errors only (default)
gtmlp.SetLogLevel(slog.LevelWarn)
// Custom handler (JSON format, custom writer, etc.)
handler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo})
gtmlp.SetLogger(slog.New(handler))Log levels:
Debug- XPath evaluation, fallback usage, field extractionInfo- HTTP requests, scraping progress, paginationWarn- Fallback usage, HTTP warnings, duplicate URLs (default)Error- HTTP failures, parsing errors, validation failures
GTMLP includes built-in SSRF (Server-Side Request Forgery) protection:
config := >mlp.Config{
Container: "//div[@class='product']",
Fields: fields,
// SSRF protection (default: enabled)
// Blocks: localhost, 127.0.0.1, 10.x.x.x, 192.168.x.x, 169.254.169.254
AllowPrivateIPs: false, // set true to allow private IPs
// Custom URL validator
URLValidator: func(url string) error {
if !strings.Contains(url, "example.com") {
return errors.New("domain not allowed")
}
return nil
},
}See SECURITY.md for security best practices.
// Load config from file
config, _ := gtmlp.LoadConfig("selectors.yaml", nil)
// Or embed with go:embed
//go:embed selectors.yaml
var yaml string
config, _ := gtmlp.ParseConfig(yaml, gtmlp.FormatYAML, nil)
// Scrape
products, _ := gtmlp.ScrapeURL[Product](context.Background(), url, config)
results, _ := gtmlp.ScrapeURLUntyped(context.Background(), url, config) // returns []map[string]anyexport GTMLP_TIMEOUT=30s
export GTMLP_USER_AGENT=MyBot/1.0
export GTMLP_PROXY=http://proxy:8080Handle varying HTML structures with altXpath and altContainer:
{
"container": "//div[@class='product']",
"altContainer": ["//article[@class='product']", "//div[@class='item']"],
"fields": {
"name": {
"xpath": ".//h2/text()",
"altXpath": [".//h3/text()", ".//h1/text()"]
},
"price": {
"xpath": ".//span[@class='price']/text()",
"altXpath": [".//div[@class='price']/text()"],
"pipes": ["trim", "tofloat"]
}
}
}How it works:
- Tries primary XPath first
- If empty (after pipes), tries each
altXpathin order - Returns first non-empty result
- Container fallback works the same way with
altContainer
Auto-follow pagination or extract URLs for manual control:
Next-Link Pagination (follow "Next" buttons):
{
"container": "//div[@class='product']",
"fields": {
"name": {"xpath": ".//h2/text()"}
},
"pagination": {
"type": "next-link",
"nextSelector": "//a[@rel='next']/@href",
"altSelectors": ["//a[contains(text(), 'Next')]/@href"],
"maxPages": 50
}
}Numbered Pagination (extract all page links):
{
"pagination": {
"type": "numbered",
"pageSelector": "//div[@class='pagination']//a/@href",
"maxPages": 20
}
}Usage:
// Auto-follow: returns combined results from all pages
products, _ := gtmlp.ScrapeURL[Product](ctx, url, config)
// Page-separated: get results per page with metadata
results, _ := gtmlp.ScrapeURLWithPages[Product](ctx, url, config)
// Extract-only: get URLs for manual control
info, _ := gtmlp.ExtractPaginationURLs(ctx, url, config)Transform extracted data using pipes:
{
"container": "//div[@class='product']",
"fields": {
"name": {"xpath": ".//h2/text()", "pipes": ["trim"]},
"price": {"xpath": ".//span[@class='price']/text()", "pipes": ["trim", "tofloat"]},
"url": {"xpath": ".//a/@href", "pipes": ["parseurl"]}
}
}Built-in pipes:
trim- Remove whitespacetoint- Convert to integer (strips$,,)tofloat- Convert to float (strips$,,)parseurl- Convert relative URLs to absoluteparsetime:layout:timezone- Parse datetimeregexreplace:pattern:replacement:flags- Regex substitutionhumanduration- Convert seconds to "X minutes ago"
Custom pipes:
gtmlp.RegisterPipe("uppercase", func(ctx context.Context, input string, params []string) (any, error) {
return strings.ToUpper(input), nil
})See docs/API_V2.md for complete pipe documentation.
- API_V2.md - Complete API reference
- examples/v2/ - 10 working examples:
- Basic scraping (JSON/YAML, embed)
- E-commerce and tables
- Pagination (next-link, numbered)
MIT