Documentation
¶
Overview ¶
Package decode turns fetched bytes into data a transform can read: JSON, YAML, XML, CSV, HTML, the Prometheus text format and plain text, after converting the body to UTF-8 from the charset it declares.
Index ¶
- Constants
- func CheckCharset(name string) error
- func ConvertedFrom(sent *fetch.HTTPResponse, c *model.Collector, kind string) string
- func JSONValue(body []byte) (value any, rest []byte, err error)
- func ParseHTML(body []byte) (*goquery.Document, error)
- func ParseXML(body []byte) (*xmlquery.Node, error)
- func TextFrom(body []byte, name string) ([]byte, error)
- type CSVColumn
- type CSVRows
- func (t *CSVRows) Cell(i int, column string) (string, bool)
- func (t *CSVRows) Columns() []CSVColumn
- func (t *CSVRows) Document() []any
- func (t *CSVRows) Has(column string) bool
- func (t *CSVRows) Len() int
- func (t *CSVRows) Longest() int
- func (t *CSVRows) MarshalJSON() ([]byte, error)
- func (t *CSVRows) Named() bool
- func (t *CSVRows) Row(i int) []string
- type Decoded
- type GraphiteReport
- type HTMLDecoded
- type PrometheusReport
Constants ¶
const KindRows = "sql"
KindRows is the Kind of the rows of a statement, which a driver read value by value and no decoder decoded; it is shown as the format of the answer, as a decoder's name is.
const MaxDepth = 10000
MaxDepth is how deep the lists and mappings of a decoded value nest at most, one inside another, whichever decoder made it: a response whose value would nest deeper is refused when it is decoded, saying so. It is encoding/json's limit on arrays and objects, which the json decoder had from the start, and one limit for every format so that what reads a value need not ask which decoder made it. What has to follow a value as deep as it goes takes the depth from here: a value is handed to a Python script, and what a script leaves is read back, to this depth (transform/pythonrequest.go, pythonanswer.go, pythonworker.go), with no limit of the hand-over's own for a response to decode within and then fail on. Each decoder is within it:
- JSON counts the arrays and objects around what it is reading, and refuses the document at one more (enter).
- YAML refuses a document whose sequences and mappings nest deeper, an alias counted as what it stands for, before anything decodes it (yamlkeys.go, yamlTooDeep). The library's parser does not keep a document within the limit: it bounds what is nested by indentation and what is nested in brackets apart, at 10,000 levels each (scannerc.go, max_indents and max_flow_level), and a level of indentation is two collections where a mapping's value is a sequence written no further in than its key, so a document without an alias parses nested up to 30,000 deep; and it bounds what aliases expand to by its size, not by its depth.
- CSV is a list of rows, each a list or a mapping of its fields; Graphite's series and a Prometheus exposition as a script reads it nest a few levels, of the exporter's own making.
- Text is text, and so are HTML and XML to a script; as documents their elements nest MaxXMLDepth deep at most.
- What a pre-script leaves in data is read by JSONValue, to this depth, and the worker refuses to write it nested deeper (pythonworker.go, deep_check).
const MaxXMLDepth = 512
MaxXMLDepth is how deep the elements of an XML document may nest. It is the bound the HTML parser has for its open elements, so XPath meets the same documents whichever decoder read them. What reads a parsed document walks it by recursion, and the text of an element is that of everything beneath it, so a document of a hundred thousand nested elements, which fits in a megabyte, costs a rule that selects them all the square of that.
Variables ¶
This section is empty.
Functions ¶
func CheckCharset ¶
CheckCharset validates response.charset when the configuration loads.
func ConvertedFrom ¶
ConvertedFrom is the encoding Decode converts the body of sent from, by its WHATWG name, for a response the collector c reads as the decoder kind: what its byte order mark, response.charset, its Content-Type or the document itself names, in Decode's own order. It is empty for a body that is not converted: one that nothing names an encoding for, or that is UTF-8 already. sent is the response as the target sent it, before Decode replaced its body.
func JSONValue ¶
JSONValue reads the JSON value body starts with as the json decoder reads a document, and returns it and what of body follows it. It is how the answer of a Python worker is read (transform/pythonanswer.go): what a pre-script leaves in data is a value inside the answer, and is read as a response's JSON is, nested MaxDepth deep at most whatever lies around it in the answer: a script may leave what it was given, as deep as a decoder made it.
func ParseHTML ¶
ParseHTML parses an HTML document as the css and xpath transforms read it: what the parser makes of the body, with the content of every template element taken out of it.
A template holds markup a script copies into the page, a row with {{name}} where a name goes. To a browser its content is no part of the document: it stands in a fragment of its own, which no selector matches and whose text is no element's. The parser here builds it as ordinary children of the template element, where `table tr` and //td found the placeholder row among the real ones, and a rule failed on a value that is "{{n}}". Rules are written against what a browser's inspector shows, so the content is detached once, here, for every reader alike: selectors, items, XPath expressions and labels do not match it, and it is no part of the text of an element around it. The template element itself stays, empty.
A template with a shadowrootmode attribute, or the shadowroot one browsers read before it, is a declarative shadow root: a browser attaches its content to the element around it and renders it, with no script, and its inspector shows it. That content is kept, as ordinary children of the template element, where selectors and expressions find it.
Types ¶
type CSVRows ¶
type CSVRows struct {
// contains filtered or unexported fields
}
CSVRows is what the csv decoder makes of a response: its rows, each the cells its line holds, and, with a header row, the header's names, once for all the rows. A row read by name has every column the header names, and those a short row lacks are empty in it, but they are not made: the decoder once gave each row a map with an entry for every column the header names, so a header of 5,000 columns over 5,000 lines of one field, a body of 39 kB, was 25 million entries and 2 GB, and a plain body of one short column held 175 times its size. The rows now cost the cells the body holds.
What a rule, an error and a script read of the rows is what they read of those maps: a row read by name as a mapping of every column the header names, the cells it lacks empty (Cell, Document), and without a header row a list of the row's fields, read by number from 1.
func (*CSVRows) Cell ¶
Cell is the cell of row i in column, and whether the row has the column: with a header, a column the header names, the cell "" where the row is short of it; without one, a column number from 1, written as a rule names it, up to the row's own length.
func (*CSVRows) Columns ¶
Columns is the columns the header names, unnamed ones left out, in the order of their names' bytes, as a script reads a row's keys; nil without a header row.
func (*CSVRows) Document ¶
Document is the rows as the decoder once gave them: a list of maps of every column the header names to its cell, "" where the row is short of it, or without a header a list of lists of fields. It costs what that cost, the header's columns times the rows, and is for json.Marshal, which writes the rows as a script reads them, and for tests.
func (*CSVRows) Has ¶
Has reports whether any row has column: with a header, one the header names, when there is a row; without one, a number up to the longest row's length.
func (*CSVRows) MarshalJSON ¶
MarshalJSON writes the rows as json.Marshal writes Document.
type Decoded ¶
type Decoded struct {
// Kind is the decoder's name, or KindRows for the rows a driver read.
Kind string
Data any
Raw []byte
// Graphite reports what the graphite decoder left out; nil for the
// others.
Graphite *GraphiteReport
// Prometheus reports the sample lines the prometheus decoder left out;
// nil when it left out none, and for the others.
Prometheus *PrometheusReport
// Columns is a sql collector's columns (fetch.HTTPResponse.Columns);
// nil for the others.
Columns []string
}
Decoded is a response decoded for a transform. Kind names the decoder, and Data holds what it produced: normalized JSON or YAML values, the same values of NDJSON, logfmt, TOML and INI (ndjson.go, logfmt.go, toml.go, ini.go), CSV rows, an *xmlquery.Node, an *HTMLDecoded, a model.MetricSet for Prometheus text, the Graphite series document (graphite.go), the body as a string, or nil for decoder none. Raw is the body the decoder read.
func Decode ¶
Decode converts r's body to UTF-8 and decodes it with the decoder c's configuration, the response's content type or its content selects. The error of a body that cannot be decoded is no longer than maxFailureBytes, whatever the body holds and whichever decoder read it (failurebound.go).
type GraphiteReport ¶
type GraphiteReport struct {
// NoPoints, Stale and Duplicates count the series left out: with no
// point that has a value, with the newest older than max_age, and
// answered twice.
NoPoints, Stale, Duplicates int
// SkippedLines counts the carbon lines skipped under invalid_lines:
// skip, and FirstSkipped says which was first and why: an error, so that
// the log takes it for the same when only the line it names differs
// (model.SameFailureText).
SkippedLines int
FirstSkipped error
}
GraphiteReport is what the graphite decoder left out of its document.
func (*GraphiteReport) LeftOut ¶
func (r *GraphiteReport) LeftOut() int
LeftOut is how many series were left out.
type HTMLDecoded ¶
HTMLDecoded is a parsed HTML document and the body it was parsed from.
type PrometheusReport ¶
type PrometheusReport struct {
// LeftOutLines counts the sample lines left out as no part of their
// histogram or summary family, in the families that are kept, and
// FirstLeftOut says which was first and what was expected in its place:
// an error, so that the log takes it for the same when only the line it
// names differs (model.SameFailureText).
LeftOutLines int
FirstLeftOut error
}
PrometheusReport is what the prometheus decoder left out of an exposition.