decode

package
v0.0.0-...-85a7ce0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 11, 2026 License: Apache-2.0 Imports: 37 Imported by: 0

Documentation

Overview

Package decode turns fetched bytes into data a transform can read: JSON, YAML, XML, CSV, HTML, the Prometheus text format and plain text, after converting the body to UTF-8 from the charset it declares.

Index

Constants

View Source
const KindRows = "sql"

KindRows is the Kind of the rows of a statement, which a driver read value by value and no decoder decoded; it is shown as the format of the answer, as a decoder's name is.

View Source
const MaxDepth = 10000

MaxDepth is how deep the lists and mappings of a decoded value nest at most, one inside another, whichever decoder made it: a response whose value would nest deeper is refused when it is decoded, saying so. It is encoding/json's limit on arrays and objects, which the json decoder had from the start, and one limit for every format so that what reads a value need not ask which decoder made it. What has to follow a value as deep as it goes takes the depth from here: a value is handed to a Python script, and what a script leaves is read back, to this depth (transform/pythonrequest.go, pythonanswer.go, pythonworker.go), with no limit of the hand-over's own for a response to decode within and then fail on. Each decoder is within it:

  • JSON counts the arrays and objects around what it is reading, and refuses the document at one more (enter).
  • YAML refuses a document whose sequences and mappings nest deeper, an alias counted as what it stands for, before anything decodes it (yamlkeys.go, yamlTooDeep). The library's parser does not keep a document within the limit: it bounds what is nested by indentation and what is nested in brackets apart, at 10,000 levels each (scannerc.go, max_indents and max_flow_level), and a level of indentation is two collections where a mapping's value is a sequence written no further in than its key, so a document without an alias parses nested up to 30,000 deep; and it bounds what aliases expand to by its size, not by its depth.
  • CSV is a list of rows, each a list or a mapping of its fields; Graphite's series and a Prometheus exposition as a script reads it nest a few levels, of the exporter's own making.
  • Text is text, and so are HTML and XML to a script; as documents their elements nest MaxXMLDepth deep at most.
  • What a pre-script leaves in data is read by JSONValue, to this depth, and the worker refuses to write it nested deeper (pythonworker.go, deep_check).
View Source
const MaxXMLDepth = 512

MaxXMLDepth is how deep the elements of an XML document may nest. It is the bound the HTML parser has for its open elements, so XPath meets the same documents whichever decoder read them. What reads a parsed document walks it by recursion, and the text of an element is that of everything beneath it, so a document of a hundred thousand nested elements, which fits in a megabyte, costs a rule that selects them all the square of that.

Variables

This section is empty.

Functions

func CheckCharset

func CheckCharset(name string) error

CheckCharset validates response.charset when the configuration loads.

func ConvertedFrom

func ConvertedFrom(sent *fetch.HTTPResponse, c *model.Collector, kind string) string

ConvertedFrom is the encoding Decode converts the body of sent from, by its WHATWG name, for a response the collector c reads as the decoder kind: what its byte order mark, response.charset, its Content-Type or the document itself names, in Decode's own order. It is empty for a body that is not converted: one that nothing names an encoding for, or that is UTF-8 already. sent is the response as the target sent it, before Decode replaced its body.

func JSONValue

func JSONValue(body []byte) (value any, rest []byte, err error)

JSONValue reads the JSON value body starts with as the json decoder reads a document, and returns it and what of body follows it. It is how the answer of a Python worker is read (transform/pythonanswer.go): what a pre-script leaves in data is a value inside the answer, and is read as a response's JSON is, nested MaxDepth deep at most whatever lies around it in the answer: a script may leave what it was given, as deep as a decoder made it.

func ParseHTML

func ParseHTML(body []byte) (*goquery.Document, error)

ParseHTML parses an HTML document as the css and xpath transforms read it: what the parser makes of the body, with the content of every template element taken out of it.

A template holds markup a script copies into the page, a row with {{name}} where a name goes. To a browser its content is no part of the document: it stands in a fragment of its own, which no selector matches and whose text is no element's. The parser here builds it as ordinary children of the template element, where `table tr` and //td found the placeholder row among the real ones, and a rule failed on a value that is "{{n}}". Rules are written against what a browser's inspector shows, so the content is detached once, here, for every reader alike: selectors, items, XPath expressions and labels do not match it, and it is no part of the text of an element around it. The template element itself stays, empty.

A template with a shadowrootmode attribute, or the shadowroot one browsers read before it, is a declarative shadow root: a browser attaches its content to the element around it and renders it, with no script, and its inspector shows it. That content is kept, as ordinary children of the template element, where selectors and expressions find it.

func ParseXML

func ParseXML(body []byte) (*xmlquery.Node, error)

ParseXML parses an XML document, and refuses one whose elements nest deeper than MaxXMLDepth.

func TextFrom

func TextFrom(body []byte, name string) ([]byte, error)

TextFrom is body, which is in the named encoding, as UTF-8 text, for showing what a target sent. Unlike Decode it changes nothing in the text: an XML declaration keeps the encoding it names.

Types

type CSVColumn

type CSVColumn struct {
	Name  string
	Place int
}

CSVColumn is a column the header names, and its place.

type CSVRows

type CSVRows struct {
	// contains filtered or unexported fields
}

CSVRows is what the csv decoder makes of a response: its rows, each the cells its line holds, and, with a header row, the header's names, once for all the rows. A row read by name has every column the header names, and those a short row lacks are empty in it, but they are not made: the decoder once gave each row a map with an entry for every column the header names, so a header of 5,000 columns over 5,000 lines of one field, a body of 39 kB, was 25 million entries and 2 GB, and a plain body of one short column held 175 times its size. The rows now cost the cells the body holds.

What a rule, an error and a script read of the rows is what they read of those maps: a row read by name as a mapping of every column the header names, the cells it lacks empty (Cell, Document), and without a header row a list of the row's fields, read by number from 1.

func (*CSVRows) Cell

func (t *CSVRows) Cell(i int, column string) (string, bool)

Cell is the cell of row i in column, and whether the row has the column: with a header, a column the header names, the cell "" where the row is short of it; without one, a column number from 1, written as a rule names it, up to the row's own length.

func (*CSVRows) Columns

func (t *CSVRows) Columns() []CSVColumn

Columns is the columns the header names, unnamed ones left out, in the order of their names' bytes, as a script reads a row's keys; nil without a header row.

func (*CSVRows) Document

func (t *CSVRows) Document() []any

Document is the rows as the decoder once gave them: a list of maps of every column the header names to its cell, "" where the row is short of it, or without a header a list of lists of fields. It costs what that cost, the header's columns times the rows, and is for json.Marshal, which writes the rows as a script reads them, and for tests.

func (*CSVRows) Has

func (t *CSVRows) Has(column string) bool

Has reports whether any row has column: with a header, one the header names, when there is a row; without one, a number up to the longest row's length.

func (*CSVRows) Len

func (t *CSVRows) Len() int

Len is how many rows there are, the header's line not counted.

func (*CSVRows) Longest

func (t *CSVRows) Longest() int

Longest is the length of the longest row.

func (*CSVRows) MarshalJSON

func (t *CSVRows) MarshalJSON() ([]byte, error)

MarshalJSON writes the rows as json.Marshal writes Document.

func (*CSVRows) Named

func (t *CSVRows) Named() bool

Named reports whether the rows are read by the header's names.

func (*CSVRows) Row

func (t *CSVRows) Row(i int) []string

Row is the cells of row i, from 0: those its line holds, which with a header are as many as the header's columns at most.

type Decoded

type Decoded struct {
	// Kind is the decoder's name, or KindRows for the rows a driver read.
	Kind string
	Data any
	Raw  []byte
	// Graphite reports what the graphite decoder left out; nil for the
	// others.
	Graphite *GraphiteReport
	// Prometheus reports the sample lines the prometheus decoder left out;
	// nil when it left out none, and for the others.
	Prometheus *PrometheusReport
	// Columns is a sql collector's columns (fetch.HTTPResponse.Columns);
	// nil for the others.
	Columns []string
}

Decoded is a response decoded for a transform. Kind names the decoder, and Data holds what it produced: normalized JSON or YAML values, the same values of NDJSON, logfmt, TOML and INI (ndjson.go, logfmt.go, toml.go, ini.go), CSV rows, an *xmlquery.Node, an *HTMLDecoded, a model.MetricSet for Prometheus text, the Graphite series document (graphite.go), the body as a string, or nil for decoder none. Raw is the body the decoder read.

func Decode

func Decode(r *fetch.HTTPResponse, c *model.Collector) (*Decoded, error)

Decode converts r's body to UTF-8 and decodes it with the decoder c's configuration, the response's content type or its content selects. The error of a body that cannot be decoded is no longer than maxFailureBytes, whatever the body holds and whichever decoder read it (failurebound.go).

type GraphiteReport

type GraphiteReport struct {
	// NoPoints, Stale and Duplicates count the series left out: with no
	// point that has a value, with the newest older than max_age, and
	// answered twice.
	NoPoints, Stale, Duplicates int
	// SkippedLines counts the carbon lines skipped under invalid_lines:
	// skip, and FirstSkipped says which was first and why: an error, so that
	// the log takes it for the same when only the line it names differs
	// (model.SameFailureText).
	SkippedLines int
	FirstSkipped error
}

GraphiteReport is what the graphite decoder left out of its document.

func (*GraphiteReport) LeftOut

func (r *GraphiteReport) LeftOut() int

LeftOut is how many series were left out.

type HTMLDecoded

type HTMLDecoded struct {
	Document *goquery.Document
	Raw      []byte
}

HTMLDecoded is a parsed HTML document and the body it was parsed from.

type PrometheusReport

type PrometheusReport struct {
	// LeftOutLines counts the sample lines left out as no part of their
	// histogram or summary family, in the families that are kept, and
	// FirstLeftOut says which was first and what was expected in its place:
	// an error, so that the log takes it for the same when only the line it
	// names differs (model.SameFailureText).
	LeftOutLines int
	FirstLeftOut error
}

PrometheusReport is what the prometheus decoder left out of an exposition.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址