data_extraction

package
v1.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 2, 2026 License: GPL-3.0 Imports: 28 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func GetRetryableHTTPClientForExchange

func GetRetryableHTTPClientForExchange(exchange string, maxRetries int) (*retryablehttp.Client, error)

func Truncate

func Truncate(r *big.Rat, prec int) string

Truncate returns r truncated to `prec` decimal places as a *big.Rat.

Types

type BinanceResponse

type BinanceResponse struct {
	Price     string `json:"lastPrice"`
	Volume    string `json:"volume"`
	Symbol    string `json:"symbol"`
	Timestamp int64  `json:"closeTime"`
}

type BitstampResponse

type BitstampResponse struct {
	Price     string `json:"last"`
	Volume    string `json:"volume"`
	Timestamp string `json:"timestamp"`
}

type BybitListItem

type BybitListItem struct {
	Price  string `json:"lastPrice"`
	Volume string `json:"volume24h"`
	Symbol string `json:"symbol"`
}

type BybitResponse

type BybitResponse struct {
	Timestamp int64 `json:"time"`
	Result    struct {
		List []BybitListItem `json:"list"`
	} `json:"result"`
}

type CoinbaseResponse

type CoinbaseResponse struct {
	Price     string `json:"price"`
	Volume    string `json:"volume"`
	Timestamp string `json:"time"`
}

type CryptoListItem

type CryptoListItem struct {
	Price     string `json:"k"`
	Volume    string `json:"v"`
	Symbol    string `json:"i"`
	Timestamp int64  `json:"t"`
}

type CryptoResponse

type CryptoResponse struct {
	Result struct {
		Data []CryptoListItem `json:"data"`
	} `json:"result"`
}

type ExchangePrice

type ExchangePrice struct {
	Exchange string `json:"exchange"` // Exchange name.
	Price    string `json:"price"`    // Price.
	Volume   string `json:"volume"`   // Volume.
	Token    string `json:"token"`    // Token.
	Symbol   string `json:"symbol"`   // Symbol.
}

ExchangePrice represents a price from a single exchange

func CalculateVolumeWeightedAverage

func CalculateVolumeWeightedAverage(prices []ExchangePrice, precision uint, token string) (string, string, int, []ExchangePrice, *appErrors.AppError)

CalculateVolumeWeightedAverage calculates the volume-weighted average price

type ExtractDataResult

type ExtractDataResult struct {
	ResponseBody    string // The response body.
	AttestationData string // The attestation data.
	StatusCode      int    // The status code.
}

ExtractDataResult represents the result of data extraction

func ExtractDataFromHTML

func ExtractDataFromHTML(ctx context.Context, attestationRequest attestation.AttestationRequest) (ExtractDataResult, *appErrors.AppError)

ExtractDataFromHTML scrapes and extracts data from HTML responses for attestation purposes.

This function: 1. Makes an HTTP request to the specified URL 2. Reads and parses the HTML content using htmlquery 3. Extracts data using XPath selectors 4. Handles different result types (element vs value) 5. Applies encoding options for numeric values 6. Returns the full HTML content and extracted data

The selector uses XPath syntax for HTML element selection: - "//div[@class='price']" - finds div with class 'price' - "//span[@id='total']/text()" - gets text content of span with id 'total' - "//table//tr[1]/td[2]" - gets second cell of first table row

HTMLResultType controls what is extracted: - "value" (default): extracts text content only - "element": extracts the complete HTML element including tags

Example usage:

request := services.AttestationRequest{
    Url: "https://example.com/price-page",
    Selector: "//span[@class='price']/text()",
    ResponseFormat: "html",
    HTMLResultType: &[]string{"value"}[0],
    EncodingOptions: encoding.EncodingOptions{Value: "float", Precision: 2}
}
result, err := ExtractDataFromHTML(request)

func ExtractDataFromJSON

func ExtractDataFromJSON(ctx context.Context, attestationRequest attestation.AttestationRequest) (ExtractDataResult, *appErrors.AppError)

ExtractDataFromJSON fetches and extracts data from JSON responses for attestation purposes.

This function: 1. Makes an HTTP request to the specified URL 2. Parses the JSON response 3. Extracts a specific value using the provided selector (supports nested paths and array indexing) 4. Applies encoding options (e.g., float precision) 5. Returns the full response body and extracted data

The selector supports dot notation and array indexing: - "data.price" - extracts price from data object - "items[0].value" - extracts value from first item in items array - "results.data[2].metrics.total" - complex nested path with array access

Example usage:

request := services.AttestationRequest{
    Url: "https://api.example.com/price",
    Selector: "data.price",
    ResponseFormat: "json",
    EncodingOptions: encoding.EncodingOptions{Value: "float", Precision: 6}
}
result, err := ExtractDataFromJSON(request)

func ExtractDataFromTargetURL

func ExtractDataFromTargetURL(ctx context.Context, attestationRequest attestation.AttestationRequest, timestamp int64) (ExtractDataResult, *appErrors.AppError)

ExtractDataFromTargetURL fetches the data from the attestation request target URL. This is the main entry point that routes to specific extractors based on the request type.

type GateResponse

type GateResponse []GateResponseItem

type GateResponseItem

type GateResponseItem struct {
	Price  string `json:"last"`
	Volume string `json:"base_volume"`
	Symbol string `json:"currency_pair"`
}

type GeminiResponse

type GeminiResponse struct {
	Price      string           `json:"last"`
	VolumeInfo GeminiVolumeInfo `json:"volume"`
}

type GeminiVolumeInfo

type GeminiVolumeInfo struct {
	USDT      string `json:"USDT,omitempty"`
	USDC      string `json:"USDC,omitempty"`
	USD       string `json:"USD,omitempty"`
	Timestamp int64  `json:"timestamp"`
}

type KrakenResponse

type KrakenResponse struct {
	Error  []string                      `json:"error"`
	Result map[string]KrakenResponseItem `json:"result"`
}

type KrakenResponseItem

type KrakenResponseItem struct {
	Price  [2]string `json:"c"`
	Volume [2]string `json:"v"`
}

type MEXCResponse

type MEXCResponse struct {
	Price     string `json:"lastPrice"`
	Volume    string `json:"volume"`
	Symbol    string `json:"symbol"`
	Timestamp int64  `json:"closeTime"`
}

type PriceFeedClient

type PriceFeedClient struct {
	// contains filtered or unexported fields
}

PriceFeedClient is the client for the price feed.

func NewPriceFeedClient

func NewPriceFeedClient() *PriceFeedClient

NewPriceFeedClient creates a new PriceFeedClient with default configurations

func (*PriceFeedClient) ExtractPriceFeedData

func (c *PriceFeedClient) ExtractPriceFeedData(ctx context.Context, attestationRequest attestation.AttestationRequest, token string, timestamp int64) (ExtractDataResult, *appErrors.AppError)

ExtractPriceFeedData handles price feed requests and always returns the volume-weighted average price (VWAP) This ensures consistent and reliable price data for oracle attestations

func (*PriceFeedClient) FetchPriceFromExchange

func (c *PriceFeedClient) FetchPriceFromExchange(ctx context.Context, exchange, token, symbol string, timestamp int64) (*ExchangePrice, *appErrors.AppError)

FetchPriceFromExchange fetches price and volume data from a specific exchange.

This function performs the following steps sequentially:

  1. Retrieves the exchange configuration for the given exchange.
  2. Replace the symbol in the endpoint template.
  3. Constructs the full URL for the API request, handling cases where the BaseURL may or may not include the protocol.
  4. Creates a retryable HTTP client for robust network requests.
  5. Builds an HTTP GET request with the provided context.
  6. Executes the HTTP request and handles any network errors.
  7. Checks the HTTP response status code for success.
  8. Reads the response body from the exchange API.
  9. Attempts to decode the response body as a JSON object. If decoding fails and the exchange is "gate.io", attempts to decode as a JSON array and adapts the data structure accordingly.
  10. Parses the price and volume from the decoded response using the appropriate exchange-specific parser.
  11. Returns an ExchangePrice struct with the parsed data, or an error if any step fails.

Parameters:

  • ctx: The context for request cancellation and logging.
  • exchange: The key identifying the exchange (e.g., "binance").
  • token: The token (e.g., "BTC").
  • symbol: The trading symbol (e.g., "BTCUSDT").

Returns:

  • *ExchangePrice: The parsed price and volume data from the exchange.
  • *appErrors.AppError: An application error if any step fails, otherwise nil.

func (*PriceFeedClient) GetPriceFeed

func (c *PriceFeedClient) GetPriceFeed(ctx context.Context, tokenName string, timestamp int64, precision uint) (*PriceFeedResult, *appErrors.AppError)

GetPriceFeed fetches and calculates the volume-weighted average price for a given token

type PriceFeedResult

type PriceFeedResult struct {
	Token              string          `json:"token"`              // Token.
	VolumeWeightedAvg  string          `json:"volumeWeightedAvg"`  // Volume-weighted average price.
	TotalVolume        string          `json:"totalVolume"`        // Total volume.
	ExchangeCount      int             `json:"exchangeCount"`      // Number of exchanges.
	Timestamp          int64           `json:"timestamp"`          // Timestamp.
	ExchangePricesRaw  []ExchangePrice `json:"exchangePricesRaw"`  // Exchange prices.
	ExchangePricesUsed []ExchangePrice `json:"exchangePricesUsed"` // Exchange prices.
	Success            bool            `json:"success"`            // Success.
}

PriceFeedResult represents the result of a price feed calculation

type XTResponse

type XTResponse struct {
	Result []XTResponseItem `json:"result"`
}

type XTResponseItem

type XTResponseItem struct {
	Price     string `json:"c"`
	Volume    string `json:"q"`
	Timestamp int64  `json:"t"`
	Symbol    string `json:"s"`
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址