xpath

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 2, 2025 License: Apache-2.0 Imports: 9 Imported by: 0

README

XPath Task

The xpath task extracts data from XML and HTML documents using XPath expressions, enabling structured data extraction from web pages and XML documents.

Function

The XPath task applies XPath queries to XML/HTML data to extract specific elements, attributes, or text content. It's particularly useful for web scraping and XML document processing.

Behavior

The XPath task extracts structured data from XML and HTML documents using XPath expressions. It receives records from its input channel, optionally selects container elements using a container XPath, then extracts multiple fields using field-specific XPath expressions. The extracted data is returned as JSON with field names as keys. The task can handle missing fields gracefully when ignore_missing is enabled.

Configuration Fields

Field Type Default Description
name string - Task name for identification
type string xpath Must be "xpath"
container string - XPath expression to select container elements
fields map[string]string - Map of field names to XPath expressions
ignore_missing bool true Whether to ignore missing fields
fail_on_error bool false Whether to stop the pipeline if this task encounters an error

Example Configurations

Extract multiple fields from HTML:
tasks:
  - name: extract_article_data
    type: xpath
    container: "//article"
    fields:
      title: ".//h1/text()"
      author: ".//span[@class='author']/text()"
      content: ".//div[@class='content']/text()"
      publish_date: ".//time/@datetime"
Extract product information:
tasks:
  - name: extract_products
    type: xpath
    container: "//div[@class='product']"
    fields:
      name: ".//h2[@class='product-name']/text()"
      price: ".//span[@class='price']/text()"
      description: ".//p[@class='description']/text()"
      image_url: ".//img/@src"
    ignore_missing: true
Extract table data:
tasks:
  - name: extract_table_data
    type: xpath
    container: "//table[@id='data-table']//tr"
    fields:
      name: ".//td[1]/text()"
      email: ".//td[2]/text()"
      phone: ".//td[3]/text()"
Extract without container (from entire document):
tasks:
  - name: extract_page_info
    type: xpath
    fields:
      title: "//title/text()"
      meta_description: "//meta[@name='description']/@content"
      canonical_url: "//link[@rel='canonical']/@href"

Sample Pipelines

  • test/pipelines/xpath.yaml - XPath extraction examples
  • test/pipelines/html2json.yaml - HTML to JSON conversion

Use Cases

  • Web scraping: Extract data from HTML web pages
  • XML processing: Parse and extract data from XML documents
  • Data extraction: Extract structured data from semi-structured sources
  • Content analysis: Analyze web page content and structure
  • API response processing: Extract data from XML API responses
  • Document processing: Parse XML-based document formats

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func New

func New() (task.Task, error)

Types

This section is empty.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址