smartcrusher

package
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 29, 2026 License: Apache-2.0 Imports: 21 Imported by: 0

Documentation

Overview

Package smartcrusher implements the SmartCrusher MVP: the JSON-array compressor that backs offloads.JsonOffload. It is a clean-room Go port of upstream chopratejas/headroom's crates/headroom-core/src/transforms/smart_crusher/*.

The package decomposes 1:1 with the upstream Rust submodules — one concern per Go file — and reuses the existing internal/ccr helpers (Store, ComputeKey, ComputeKeyMD5, the three marker builders) and internal/relevance (Scorer, NewHybridScorer). smartcrusher.SmartCrusher implements the offloads.Crusher seam; internally it carries a richer 4-field types.CrushResult, and the seam boundary maps crushed->Compressed / wasModified->WasModified while dropping Strategy/Original. All JSON objects on the compression path are decoded with github.com/iancoleman/orderedmap: map[string]any is BANNED because key insertion/parse order is load-bearing throughout.

Decision B — NO byte-parity. Upstream's stated goal is byte-exact parity with a Python reference; headroom-go REJECTS that goal. We capture behavior, ratio bounds, and marker/strategy SHAPES, not byte-exact compressed output. The four upstream Python bugs are ANTI-REQUIREMENTS — this port implements the CORRECT behavior instead:

BUG#1 percentile: proper linear-interpolation percentile, not int-div index.
BUG#2 sequential: zero-padded numeric strings (["001".."010"]) are categorical.
BUG#3 rare-status: no >10-distinct short-circuit; cardinality cap is 50.
BUG#4 k-split: kFirst+kLast clamped to <= kTotal (respects max_items_after_crush).

The only byte-pinned identifiers we replicate exactly are enum String() values, ERROR_KEYWORDS, hash hex, and marker structural literals.

Index

Constants

This section is empty.

Variables

View Source
var ErrorKeywords = []string{
	"error", "exception", "failed", "failure", "critical", "fatal",
	"crash", "panic", "abort", "timeout", "denied", "rejected",
}

ErrorKeywords is the fallback error-preservation signal: an item whose lowercased JSON contains any of these substrings is kept. The list is lowercase by construction; the CALLER lowercases the haystack and does plain substring containment (not whole-word/prefix/regex), biasing toward over-preservation. The rustdoc inside constraints.rs wrongly lists 11 (omitting "failure"); the authoritative count is 12 [ref: error_keywords.rs].

Functions

func CalculateStringEntropy

func CalculateStringEntropy(s string) float64

CalculateStringEntropy returns the normalized Shannon entropy over RUNES (code points), in [0,1]. Strings shorter than two runes, or with a single distinct rune, return 0.0 (maxEntropy log2(1)=0 is guarded) [ref: statistics.rs algorithm]. Rune-level is correct here (a statistical feature, not byte math).

func DeduplicateIndicesByContent

func DeduplicateIndicesByContent(keepIndices map[int]struct{}, items []any) map[int]struct{}

DeduplicateIndicesByContent collapses content-duplicate indices to the lowest index [ref: orchestration.rs deduplicate_indices_by_content]. Iterating the input set ascending, it records the first index seen for each content hash, skipping out-of-bounds indices. An empty input yields an empty set.

func DetectErrorItemsForPreservation

func DetectErrorItemsForPreservation(items []*orderedmap.OrderedMap, itemStrings []string) []int

DetectErrorItemsForPreservation returns the ascending indices of object items whose lowercased JSON representation contains any ErrorKeywords substring [ref: outliers.rs detect_error_items_for_preservation]. When itemStrings is provided and long enough, the cached string for an index is used verbatim (lowercased); otherwise the item is freshly serialized. Non-object (nil) elements are skipped BEFORE the cache lookup.

func DetectIDFieldStatistically

func DetectIDFieldStatistically(stats FieldStats, values []any) (bool, float64)

DetectIDFieldStatistically reports whether the field is an identifier column and, if so, a confidence in [0,0.95]. The UniqueRatio<0.9 hard gate short- circuits first; then string (UUID / high-entropy) and numeric (sequential / range) branches, then a >0.98-uniqueness catch-all [ref: field_detect.rs detect_id]. All uniqueness comparisons are strict.

func DetectRareStatusValues

func DetectRareStatusValues(items []*orderedmap.OrderedMap, commonFields map[string]struct{}) []int

DetectRareStatusValues returns the indices of items whose value on a common field falls OUTSIDE that field's Pareto-dominant cluster [ref: outliers.rs detect_rare_status_values]. The result is APPEND-ordered (scan order) and may contain duplicate indices across fields; the caller (DetectStructuralOutliers) deduplicates. A field qualifies only when its distinct non-null values number between 2 and 50 inclusive and no more than 5 values are needed to reach 80% coverage.

func DetectScoreFieldStatistically

func DetectScoreFieldStatistically(stats FieldStats, items []any) (bool, float64)

DetectScoreFieldStatistically reports whether the field is a score/relevance column and, if so, a confidence in [0,0.95]. It requires a numeric field with both min and max set, accumulates range/descending/float bonuses, and rejects sequential columns [ref: field_detect.rs detect_score]. The [-1,1] range check is intentionally ASYMMETRIC (min>=-1 AND max<=1) — do NOT symmetrize.

func DetectSequentialPattern

func DetectSequentialPattern(values []any, checkOrder bool) bool

DetectSequentialPattern reports whether values form a near-arithmetic sequence with a unit-ish step (avg diff in [0.5,2.0]). BUG#2 fixed: purely-string numerics are treated as categorical and rejected — at least one real JSON Number must be present [ref: statistics.rs algorithm, BUG#2]. When checkOrder is set, the ORIGINAL (unsorted) order must be mostly ascending too.

func DetectStructuralOutliers

func DetectStructuralOutliers(items []*orderedmap.OrderedMap) []int

DetectStructuralOutliers returns the ascending-sorted, deduplicated indices of items that look structurally anomalous: items carrying a rare field, plus items carrying a rare status value on a common field [ref: outliers.rs detect_structural_outliers]. Non-object (nil) elements are skipped in field scans but count toward the denominator n, which shifts the common/rare thresholds. An array shorter than 5 yields no outliers.

func ExtractQueryAnchors

func ExtractQueryAnchors(text string) map[string]struct{}

ExtractQueryAnchors returns the lowercased anchor set for text. The set is a map[string]struct{} because order is irrelevant on the query side (this is not the compression path). Empty text yields an empty set. Each regex runs over the FULL original text independently, so a substring may contribute to multiple categories [ref: anchors.rs extract_query_anchors].

func FillRemainingSlots

func FillRemainingSlots(keepIndices map[int]struct{}, items []any, n, effectiveMax int) map[int]struct{}

FillRemainingSlots tops the keep set up toward effectiveMax with content-unique, stride-sampled candidates [ref: orchestration.rs fill_remaining_slots]. When the set is already at/over budget it is returned unchanged. Candidates are the in-bounds indices NOT already kept, walked in a stride pattern so the fill is spread across the array rather than clustered at the head.

func HashFieldName

func HashFieldName(fieldName string) string

HashFieldName returns the first 8 lowercase-hex chars of the SHA-256 digest of the field name's UTF-8 bytes. It mirrors Python hashlib.sha256(name.encode()).hexdigest()[:8] and is used as a cache key to look up TOIN-anonymized preserve_fields (TOIN stores names as SHA-256[:8]).

This is NOT a CCR marker hash and never emits <<ccr:...>>. The truncation is load-bearing: it MUST be [:8] — a prior [:16] silently broke all preserve-field lookups. Hash the bytes, not the runes ("café" -> "850f7dc4") [ref: hashing.rs].

func IsUUIDFormat

func IsUUIDFormat(value string) bool

IsUUIDFormat reports whether value is a canonical 36-byte 8-4-4-4-12 hex UUID. Length is measured in BYTES (Rust .len()), hex is case-insensitive, and no version/variant bits are validated [ref: statistics.rs algorithm].

func ItemMatchesAnchors

func ItemMatchesAnchors(item *orderedmap.OrderedMap, anchors map[string]struct{}) bool

ItemMatchesAnchors reports whether the Python-repr of item (lowercased) contains any anchor as a substring [ref: anchors.rs item_matches_anchors]. An empty anchor set never matches. pythonRepr includes single-quoted object keys, so matching tests BOTH keys and values.

func PrioritizeIndices

func PrioritizeIndices(cfg *Config, keepIndices map[int]struct{}, items []any, n int, analysis *ArrayAnalysis, effectiveMax int) map[int]struct{}

PrioritizeIndices resolves the final keep set under a budget [ref: orchestration.rs prioritize_indices]. It dedups (when enabled), fills toward the budget, and returns early if the result is within budget. Over budget, it keeps ALL critical rows (errors, structural outliers, numeric anomalies) plus the first-3/last-2 anchors plus remaining fill — the result MAY exceed effectiveMax because criticals are never dropped. analysis may be nil.

Types

type AnchorConfig

type AnchorConfig struct {
	FirstAnchors int
	LastAnchors  int
}

AnchorConfig holds the head/tail anchor counts [ref: (d) SAMPLING first_anchors = 3, last_anchors = 2].

func DefaultAnchorConfig

func DefaultAnchorConfig() AnchorConfig

DefaultAnchorConfig returns the MVP anchor counts: 3 leading, 2 trailing.

type AnchorPattern

type AnchorPattern int

AnchorPattern names the array shape the selector is anchoring for. Only Generic is exercised in the MVP; the other variants exist so planning.go's mapToAnchorPattern (Task 13) has a stable target set.

const (
	// AnchorGeneric is the default/fallback pattern (SmartSample -> Generic).
	AnchorGeneric AnchorPattern = iota
	// AnchorTimeSeries anchors time-ordered arrays.
	//
	// DEFERRED (Plan 4+): no distinct MVP anchoring; treated as Generic.
	AnchorTimeSeries
	// AnchorSearchResults anchors ranked search-result arrays (from TopN).
	//
	// DEFERRED (Plan 4+): no distinct MVP anchoring; treated as Generic.
	AnchorSearchResults
	// AnchorLogs anchors clustered log arrays (from ClusterSample).
	//
	// DEFERRED (Plan 4+): no distinct MVP anchoring; treated as Generic.
	AnchorLogs
)

type AnchorSelector

type AnchorSelector struct {
	// contains filtered or unexported fields
}

AnchorSelector selects anchor indices per its AnchorConfig.

func NewAnchorSelector

func NewAnchorSelector(cfg AnchorConfig) *AnchorSelector

NewAnchorSelector builds a selector from cfg.

func (*AnchorSelector) SelectAnchors

func (s *AnchorSelector) SelectAnchors(items []*orderedmap.OrderedMap, maxItems int, pattern AnchorPattern, query string) []int

SelectAnchors returns the ascending, deduplicated, in-bounds set of anchor indices for items: the first min(FirstAnchors, n), the last LastAnchors, and every index whose item matches a query anchor. maxItems and pattern are accepted for signature parity with the DEFERRED per-pattern selectors but do not affect the MVP Generic selection. An empty array yields an empty set.

type ArrayAnalysis

type ArrayAnalysis struct {
	ItemCount           int
	FieldStats          *orderedmap.OrderedMap
	DetectedPattern     string
	RecommendedStrategy CompressionStrategy
	ConstantFields      *orderedmap.OrderedMap
	EstimatedReduction  float64
	Crushability        *CrushabilityAnalysis
}

ArrayAnalysis is the full analysis of one array. FieldStats/ConstantFields are ordered maps (key order load-bearing). DetectedPattern domain: time_series|logs|search_results|generic.

type ArrayType

type ArrayType int

ArrayType is the coarse element-homogeneity label ClassifyArray assigns to a JSON array. It is a pure value classifier — no CCR markers, no sampling. Only DictArray drives an MVP strategy (the lossless compaction-table path); the other labels' downstream crushers are DEFERRED (Plan 4+), but every label is emitted so routing sees the true shape.

const (
	DictArray ArrayType = iota
	StringArray
	NumberArray
	BoolArray
	NestedArray
	MixedArray
	Empty
)

func ClassifyArray

func ClassifyArray(items []any) ArrayType

ClassifyArray labels an array by its element types. It walks EVERY element (no sampling, no early break) and sets six presence flags, then applies the homogeneity decision. A single non-conforming element (including one null among objects, or a bool mixed with numbers) collapses the array to MixedArray. There is no pure-null gate, so [null, null] -> MixedArray.

Elements are expected in the shapes decodeJSON produces: *orderedmap.OrderedMap for objects, []any for arrays, json.Number for numbers, and string/bool/nil for the remaining scalars. The Go type switch separates bool from number for us (like Rust serde), so [true, false, 1] falls through to MixedArray without any special-casing.

func (ArrayType) AsStr

func (t ArrayType) AsStr() string

AsStr returns the parity-pinned lowercase identifier for the type. These literals are byte-pinned (Decision B exception) — do NOT relax them.

type CompressionPlan

type CompressionPlan struct {
	Strategy       CompressionStrategy
	KeepIndices    []int
	ConstantFields *orderedmap.OrderedMap
	SummaryRanges  []SummaryRange
	ClusterField   *string
	SortField      *string
	KeepCount      int
}

CompressionPlan describes how to compress one array. ClusterField/SortField are nil in the MVP; SummaryRanges is empty.

func DefaultCompressionPlan

func DefaultCompressionPlan() CompressionPlan

DefaultCompressionPlan returns a None-strategy plan with empty indices/ranges, nil cluster/sort fields, and the parity-pinned KeepCount of 10.

type CompressionStrategy

type CompressionStrategy int

CompressionStrategy enumerates the recommended crush strategies. In the MVP only None/Skip/SmartSample are actively assigned; TimeSeries/ClusterSample/TopN are defined-but-unassigned parity variants.

const (
	StrategyNone CompressionStrategy = iota
	StrategySkip
	StrategyTimeSeries
	StrategyClusterSample
	StrategyTopN
	StrategySmartSample
)

func (CompressionStrategy) String

func (s CompressionStrategy) String() string

String returns the parity-pinned lowercase identifier (byte-pinned; Decision B does NOT relax these). Note StrategyClusterSample is "cluster", NOT "cluster_sample".

type Config

type Config struct {
	Enabled                 bool
	MinItemsToAnalyze       int
	MinTokensToCrush        int
	VarianceThreshold       float64
	UniquenessThreshold     float64
	SimilarityThreshold     float64
	MaxItemsAfterCrush      int
	PreserveChangePoints    bool
	FactorOutConstants      bool
	IncludeSummaries        bool
	UseFeedbackHints        bool
	ToinConfidenceThreshold float64
	DedupIdenticalItems     bool
	FirstFraction           float64
	LastFraction            float64
	RelevanceThreshold      float64
	// LosslessMinSavingsRatio is the savings gate used in crushArray:
	// savings = 1 - len(rendered)/len(input) (BYTES); savings < this ratio falls
	// through to the lossy path. Upstream diverges: config.rs lowered the default
	// to 0.15, but crusher.rs docs, the parity fixtures, and this plan's brief use
	// 0.30. The MVP-effective default is 0.30; tests override to 0.99 to force the
	// lossy path.
	LosslessMinSavingsRatio float64
	EnableCCRMarker         bool

	CompactionCoreFieldFraction      float64
	CompactionHeterogeneousCoreRatio float64
	CompactionMaxFlattenInnerKeys    int
	CompactionMinBuckets             int
	CompactionMaxBuckets             int
}

Config holds the SmartCrusher tuning knobs. It is a schema-preserving compressor: output contains only items from the original array. Fields tagged DEFERRED-consumer below are defined for parity but not yet exercised in the MVP.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns the canonical SmartCrusher defaults [ref: (d) constants].

type Constraint

type Constraint interface {
	Name() string
	MustKeep(items []*orderedmap.OrderedMap, itemStrings []string) []int
}

Constraint yields the in-bounds item indices that MUST be kept. The allocator takes the union across the registered constraint stack. itemStrings is an optional pre-serialized JSON cache (nil when absent); it is a perf hint only and never changes the result. Implementations are pure and O(n) and must never return an out-of-bounds index (the allocator would silently drop it anyway).

type CrushArrayResult

type CrushArrayResult struct {
	Items          []any
	StrategyInfo   string
	CcrHash        *string
	DroppedSummary string
	Compacted      *string
	CompactionKind *string
}

CrushArrayResult is the internal result of crushArray [ref: crusher.rs CrushArrayResult]. Items is the (possibly reduced) kept slice; Compacted, when non-nil, is the rendered lossless-table string that REPLACES the array wholesale; CcrHash/DroppedSummary describe a lossy row drop (space-form marker + store hash).

type CrushEvent

type CrushEvent struct {
	Strategy    string
	InputBytes  int
	OutputBytes int
	ElapsedNs   uint64
	WasModified bool
}

CrushEvent is the per-Crush telemetry record consumed by Observers. All byte fields are BYTE lengths (Go len), NOT rune counts [ref: traits.rs].

DEFERRED (Plan 4+): only TracingObserver consumes this today; the seam carries no observer wiring, so the compression path never populates or emits it.

type CrushResult

type CrushResult struct {
	Compressed  string
	Original    string
	WasModified bool
	Strategy    string
}

CrushResult is the INTERNAL 4-field result the crusher computes. It is RICHER than the offloads.CrushResult seam type (2 fields) and stays internal — do NOT widen the seam. Conflating the two is the #1 porting error.

func PassthroughResult

func PassthroughResult(content string) CrushResult

PassthroughResult builds a no-op CrushResult (Compressed==Original==content, WasModified false, Strategy "passthrough").

type CrushabilityAnalysis

type CrushabilityAnalysis struct {
	Crushable           bool
	Confidence          float64
	Reason              string
	SignalsPresent      []string
	SignalsAbsent       []string
	HasIDField          bool
	IDUniqueness        float64
	AvgStringUniqueness float64
	HasScoreField       bool
	ErrorItemCount      int
	AnomalyCount        int
}

CrushabilityAnalysis is the analyzer's verdict for one array.

func SkipAnalysis

func SkipAnalysis(reason string, confidence float64) CrushabilityAnalysis

SkipAnalysis builds a not-crushable CrushabilityAnalysis with metrics zeroed and slices EMPTY but non-nil.

type FieldStats

type FieldStats struct {
	Name          string
	FieldType     string
	Count         int
	UniqueCount   int
	UniqueRatio   float64
	IsConstant    bool
	ConstantValue *any
	MinVal        *float64
	MaxVal        *float64
	MeanVal       *float64
	Variance      *float64
	ChangePoints  []int
	AvgLength     *float64
	TopValues     []TopValue
}

FieldStats captures per-field statistics. Option<T> upstream fields map to Go pointers so absent (nil) and zero (&0.0) stay distinct. FieldType domain: numeric|string|boolean|object|array|null.

type KeepErrorsConstraint

type KeepErrorsConstraint struct{}

KeepErrorsConstraint preserves items whose JSON contains an error keyword. It USES the itemStrings cache when provided (perf only; the result is identical to the nil path) [ref: constraints.rs keep_errors].

func (KeepErrorsConstraint) MustKeep

func (KeepErrorsConstraint) MustKeep(items []*orderedmap.OrderedMap, itemStrings []string) []int

MustKeep delegates to DetectErrorItemsForPreservation, forwarding the optional itemStrings cache.

func (KeepErrorsConstraint) Name

Name reports the stable constraint identifier.

type KeepStructuralOutliersConstraint

type KeepStructuralOutliersConstraint struct{}

KeepStructuralOutliersConstraint preserves items that look structurally anomalous (rare field or rare status value). It IGNORES itemStrings (upstream param is _item_strings) [ref: constraints.rs keep_structural_outliers].

func (KeepStructuralOutliersConstraint) MustKeep

MustKeep delegates to DetectStructuralOutliers; the itemStrings argument is deliberately unused.

func (KeepStructuralOutliersConstraint) Name

Name reports the stable constraint identifier.

type Observer

type Observer interface {
	Name() string
	OnEvent(event *CrushEvent)
}

Observer receives one CrushEvent per top-level Crush, synchronously, after the result is computed and before it is returned. Observers fire in registration order and MUST NOT panic (catch any I/O error internally).

DEFERRED (Plan 4+): telemetry only; the MVP seam does not plumb observers.

type Scorer

type Scorer = relevance.Scorer

Scorer is the relevance scoring contract. It is an ALIAS of relevance.Scorer, not a redefinition — do NOT define a second Scorer interface [ref: traits.rs Scorer re-export].

type SmartAnalyzer

type SmartAnalyzer struct {
	// contains filtered or unexported fields
}

SmartAnalyzer is a stateless verdict engine carrying only its config.

func NewSmartAnalyzer

func NewSmartAnalyzer(config Config) *SmartAnalyzer

NewSmartAnalyzer builds an analyzer bound to config. Only VarianceThreshold and MinItemsToAnalyze are consulted here.

func (*SmartAnalyzer) AnalyzeArray

func (a *SmartAnalyzer) AnalyzeArray(items []*orderedmap.OrderedMap) ArrayAnalysis

AnalyzeArray is the top entry point [ref: analyzer.rs AnalyzeArray]. It guards empty/first-not-object arrays into a None-strategy, nil-crushability result (generic pattern), otherwise builds the sorted-key FieldStats map, detects the pattern, factors constant fields, runs the crushability tree, selects a strategy, and estimates the reduction.

func (*SmartAnalyzer) AnalyzeCrushability

func (a *SmartAnalyzer) AnalyzeCrushability(items []*orderedmap.OrderedMap, fieldStats *orderedmap.OrderedMap) CrushabilityAnalysis

AnalyzeCrushability is the six-case decision tree [ref: analyzer.rs AnalyzeCrushability]. Field iteration is ASCII-sorted throughout; case order is load-bearing (CASE0 precedes CASE1). It gathers signals (id/score/outliers/ error/anomalies/change_points), computes uniqueness aggregates, then returns the first matching verdict.

func (*SmartAnalyzer) AnalyzeField

func (a *SmartAnalyzer) AnalyzeField(key string, items []*orderedmap.OrderedMap) FieldStats

AnalyzeField computes per-field statistics [ref: analyzer.rs AnalyzeField]. Missing keys and explicit nulls both count as null values.

func (*SmartAnalyzer) DetectChangePoints

func (a *SmartAnalyzer) DetectChangePoints(values []float64, window int) []int

DetectChangePoints returns the deduped, ascending indices where the local mean shifts by more than variance_threshold*stdev [ref: analyzer.rs DetectChangePoints]. Series shorter than window*2, or with non-positive stdev, yield no change points. Subtraction i-window is saturating.

func (*SmartAnalyzer) DetectPattern

func (a *SmartAnalyzer) DetectPattern(fieldStats *orderedmap.OrderedMap, items []*orderedmap.OrderedMap) string

DetectPattern classifies the array as time_series | logs | search_results | generic [ref: analyzer.rs DetectPattern]. Field iteration is ASCII-sorted.

func (*SmartAnalyzer) DetectTemporalField

func (a *SmartAnalyzer) DetectTemporalField(fieldStats *orderedmap.OrderedMap, items []*orderedmap.OrderedMap) bool

DetectTemporalField reports whether any field looks like a timestamp [ref: analyzer.rs DetectTemporalField]. String fields sample the first 10 items and check the ISO-datetime/date fraction over the SAMPLED string count (not the full array length); numeric fields check ONLY min_val against the unix-seconds/millis ranges (max_val must merely be present).

func (*SmartAnalyzer) EstimateReduction

func (a *SmartAnalyzer) EstimateReduction(fieldStats *orderedmap.OrderedMap, strategy CompressionStrategy, itemCount int) float64

EstimateReduction estimates the fractional size reduction [ref: analyzer.rs EstimateReduction]. itemCount is unused (kept for signature parity). None strategy or empty field_stats yield 0.0 (the empty guard also prevents an upstream ZeroDivisionError).

func (*SmartAnalyzer) SelectStrategy

func (a *SmartAnalyzer) SelectStrategy(fieldStats *orderedmap.OrderedMap, pattern string, itemCount int, crushability *CrushabilityAnalysis) CompressionStrategy

SelectStrategy maps the crushability verdict + pattern to a CompressionStrategy [ref: analyzer.rs SelectStrategy]. TimeSeries/ClusterSample/TopN are computed for parity even though their crushers are DEFERRED; downstream falls back.

type SmartCrusher

type SmartCrusher struct {
	// contains filtered or unexported fields
}

SmartCrusher is the assembled crusher [ref: crusher.rs from_parts]. The struct and its fromParts constructor live here (in builder.go) because Build() needs to materialize it; the crush behavior (the offloads.Crusher seam impl, the 3-tier funnel, processValue, and the two public NewSmartCrusher* constructors) is added in crusher.go (Task 14). The planner field is derived inside fromParts from the other collaborators.

func NewSmartCrusher

func NewSmartCrusher(cfg Config) *SmartCrusher

NewSmartCrusher builds the lossless-first OSS default crusher: the two default constraints + tracing observer, the CSV-schema compaction stage (Tier-2), and an in-memory CCR store for the lossy Tier-3 offloads. The blank import in builder.go guarantees the in-memory backend is registered, so Build() cannot fail.

func NewSmartCrusherWithoutCompaction

func NewSmartCrusherWithoutCompaction(cfg Config) *SmartCrusher

NewSmartCrusherWithoutCompaction builds a lossy-only crusher (no compaction stage), still with an in-memory CCR store for the space-form lossy offloads. It forces every crushable array past the lossless tier straight to the lossy funnel.

func (*SmartCrusher) CcrGet

func (c *SmartCrusher) CcrGet(hash string) (string, bool)

CcrGet retrieves a stored CCR payload by hash (test/round-trip helper). It returns ("", false) when the store is nil or the hash is absent.

func (*SmartCrusher) CcrLen

func (c *SmartCrusher) CcrLen() int

CcrLen reports the CCR store entry count (0 when the store is nil).

func (*SmartCrusher) CompactDocumentJSON

func (c *SmartCrusher) CompactDocumentJSON(doc string) (string, error)

CompactDocumentJSON drives the DocumentCompactor walker path with the crusher's ccrStore, so opaque blobs it emits ARE store-written and round-trippable (BLAKE3 24-hex). It decodes doc, walks it, and re-serializes the result compact. Used by the parity round-trip fixtures for the walker opaque path.

func (*SmartCrusher) Crush

func (c *SmartCrusher) Crush(content, query string, bias float64) offloads.CrushResult

Crush is the offloads.Crusher seam entry point [ref: crusher.rs Crush]. It runs the crush and maps the INTERNAL result onto the 2-field seam type, dropping the Strategy/Original fields (do NOT widen the seam).

func (*SmartCrusher) CrushArrayJSON

func (c *SmartCrusher) CrushArrayJSON(arrayJSON, queryContext string, bias float64) (CrushArrayResult, error)

CrushArrayJSON is a test helper: decode a JSON array string, run crushArray, and return the raw CrushArrayResult.

type SmartCrusherBuilder

type SmartCrusherBuilder struct {
	// contains filtered or unexported fields
}

SmartCrusherBuilder is the fluent composition point for a SmartCrusher [ref: builder.rs]. It carries zero compression logic — it only assembles the config, scorer, constraint/observer stacks, optional compaction stage, and optional CCR store, then wires them into a SmartCrusher via from_parts on Build(). A fresh builder is EMPTY (no scorer/constraints/observers/compaction/store); Build() defaults only the scorer (to HybridScorer). Setters return the builder for chaining and COMPOSE (append), never replace.

func NewSmartCrusherBuilder

func NewSmartCrusherBuilder(config Config) *SmartCrusherBuilder

NewSmartCrusherBuilder returns an EMPTY builder holding config by value.

func (*SmartCrusherBuilder) AddConstraint

func (b *SmartCrusherBuilder) AddConstraint(c Constraint) *SmartCrusherBuilder

AddConstraint appends a must-keep constraint (order preserved for determinism).

func (*SmartCrusherBuilder) AddDefaultOSSConstraints

func (b *SmartCrusherBuilder) AddDefaultOSSConstraints() *SmartCrusherBuilder

AddDefaultOSSConstraints appends the OSS-default constraint stack [KeepErrors, KeepStructuralOutliers] in that order (composes, never replaces).

func (*SmartCrusherBuilder) AddObserver

AddObserver appends an observer (fires in registration order).

func (*SmartCrusherBuilder) AnchorConfig

AnchorConfig sets the anchor head/tail counts (defaulted at Build() when unset).

func (*SmartCrusherBuilder) Build

func (b *SmartCrusherBuilder) Build() *SmartCrusher

Build wires the assembled parts into a SmartCrusher [ref: builder.rs Build]. It returns NO error (builder.rs goPortNotes: defaults can't fail). It defaults the anchor config and scorer when unset, clones the config for the analyzer, and hands everything to from_parts.

func (*SmartCrusherBuilder) WithCCRStore

func (b *SmartCrusherBuilder) WithCCRStore(store ccr.Store) *SmartCrusherBuilder

WithCCRStore sets an explicit CCR store.

func (*SmartCrusherBuilder) WithCompaction

WithCompaction sets the compaction stage (run before the lossy path).

func (*SmartCrusherBuilder) WithDefaultCCRStore

func (b *SmartCrusherBuilder) WithDefaultCCRStore() *SmartCrusherBuilder

WithDefaultCCRStore sets an in-memory CCR store (1000 entries / 5-min TTL). The blank import above guarantees the in-memory backend is registered, so store construction cannot fail; mustInMemoryStore panics only on the structurally impossible error.

func (*SmartCrusherBuilder) WithDefaultCompaction

func (b *SmartCrusherBuilder) WithDefaultCompaction() *SmartCrusherBuilder

WithDefaultCompaction sets the OSS-default CSV-schema compaction stage.

func (*SmartCrusherBuilder) WithDefaultOSSSetup

func (b *SmartCrusherBuilder) WithDefaultOSSSetup() *SmartCrusherBuilder

WithDefaultOSSSetup wires the OSS defaults: HybridScorer, the two default constraints, and the TracingObserver. It equals SmartCrusher::new(config).

func (*SmartCrusherBuilder) WithScorer

func (b *SmartCrusherBuilder) WithScorer(scorer Scorer) *SmartCrusherBuilder

WithScorer plugs in a relevance scorer (enterprise seam). Build() falls back to HybridScorer when this is unset.

type SmartCrusherPlanner

type SmartCrusherPlanner struct {
	Config         Config
	AnchorSelector *AnchorSelector
	Scorer         Scorer
	Analyzer       *SmartAnalyzer
	Constraints    []Constraint
}

SmartCrusherPlanner turns an ArrayAnalysis into a CompressionPlan whose keep_indices is a sorted list of original indices to retain [ref: planning.rs]. It emits NO CCR markers and does no byte rewriting (downstream concerns). The MVP covers the create_plan dispatcher (Skip short-circuit + SmartSample routing), plan_smart_sample, and the shared signal helpers; the heavy per-pattern planners (TopN/ClusterSample/TimeSeries) are DEFERRED and route to planSmartSample.

func NewSmartCrusherPlanner

func NewSmartCrusherPlanner(config Config, anchorSelector *AnchorSelector, scorer Scorer, analyzer *SmartAnalyzer, constraints []Constraint) *SmartCrusherPlanner

NewSmartCrusherPlanner assembles a planner from its collaborators.

func (*SmartCrusherPlanner) CreatePlan

func (p *SmartCrusherPlanner) CreatePlan(analysis *ArrayAnalysis, items []*orderedmap.OrderedMap, queryContext string, preserveFields []string, effectiveMaxItems *int, itemStrings []string) *CompressionPlan

CreatePlan dispatches on analysis.RecommendedStrategy [ref: planning.rs create_plan]. effectiveMaxItems overrides config.MaxItemsAfterCrush when non-nil. A Skip verdict short-circuits to keep-all (0..n) before any dispatch. All non-Skip strategies route through planSmartSample in the MVP; the heavy planners are DEFERRED. preserveFields and itemStrings are TOIN/perf hints (callers pass nil).

type SummaryRange

type SummaryRange struct {
	Start   int
	End     int
	Summary any
}

SummaryRange is a summarized index span. Summary is *any-typed to distinguish absent from a zero value; SummaryRanges is empty in the MVP.

type TopValue

type TopValue struct {
	Value string
	Count int
}

TopValue is one entry of a field's frequency-descending top-values list.

type TracingObserver

type TracingObserver struct{}

TracingObserver is the OSS-default Observer: a stateless debug-log emitter [ref: observer.rs].

DEFERRED (Plan 4+): observability only, MVP-irrelevant to compression. The seam never plumbs observers, so OnEvent is not called on the compression path today; it is defined so WithDefaultOSSSetup can register the one default observer.

func (TracingObserver) Name

func (TracingObserver) Name() string

Name reports the stable observer identifier.

func (TracingObserver) OnEvent

func (TracingObserver) OnEvent(event *CrushEvent)

OnEvent debug-logs the crush event. It is zero-cost when debug logging is filtered (the level gate runs before any field evaluation) and MUST NOT panic — a nil event is tolerated.

Directories

Path Synopsis
Package compaction implements the lossless compaction-table stage of the SmartCrusher: it turns a homogeneous JSON array-of-objects into a compact CSV-schema (or JSON/markdown-kv) rendering, replacing bulky opaque leaves with CCR cell markers.
Package compaction implements the lossless compaction-table stage of the SmartCrusher: it turns a homogeneous JSON array-of-objects into a compact CSV-schema (or JSON/markdown-kv) rendering, replacing bulky opaque leaves with CCR cell markers.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址