Documentation
¶
Overview ¶
Package smartcrusher implements the SmartCrusher MVP: the JSON-array compressor that backs offloads.JsonOffload. It is a clean-room Go port of upstream chopratejas/headroom's crates/headroom-core/src/transforms/smart_crusher/*.
The package decomposes 1:1 with the upstream Rust submodules — one concern per Go file — and reuses the existing internal/ccr helpers (Store, ComputeKey, ComputeKeyMD5, the three marker builders) and internal/relevance (Scorer, NewHybridScorer). smartcrusher.SmartCrusher implements the offloads.Crusher seam; internally it carries a richer 4-field types.CrushResult, and the seam boundary maps crushed->Compressed / wasModified->WasModified while dropping Strategy/Original. All JSON objects on the compression path are decoded with github.com/iancoleman/orderedmap: map[string]any is BANNED because key insertion/parse order is load-bearing throughout.
Decision B — NO byte-parity. Upstream's stated goal is byte-exact parity with a Python reference; headroom-go REJECTS that goal. We capture behavior, ratio bounds, and marker/strategy SHAPES, not byte-exact compressed output. The four upstream Python bugs are ANTI-REQUIREMENTS — this port implements the CORRECT behavior instead:
BUG#1 percentile: proper linear-interpolation percentile, not int-div index. BUG#2 sequential: zero-padded numeric strings (["001".."010"]) are categorical. BUG#3 rare-status: no >10-distinct short-circuit; cardinality cap is 50. BUG#4 k-split: kFirst+kLast clamped to <= kTotal (respects max_items_after_crush).
The only byte-pinned identifiers we replicate exactly are enum String() values, ERROR_KEYWORDS, hash hex, and marker structural literals.
Index ¶
- Variables
- func CalculateStringEntropy(s string) float64
- func DeduplicateIndicesByContent(keepIndices map[int]struct{}, items []any) map[int]struct{}
- func DetectErrorItemsForPreservation(items []*orderedmap.OrderedMap, itemStrings []string) []int
- func DetectIDFieldStatistically(stats FieldStats, values []any) (bool, float64)
- func DetectRareStatusValues(items []*orderedmap.OrderedMap, commonFields map[string]struct{}) []int
- func DetectScoreFieldStatistically(stats FieldStats, items []any) (bool, float64)
- func DetectSequentialPattern(values []any, checkOrder bool) bool
- func DetectStructuralOutliers(items []*orderedmap.OrderedMap) []int
- func ExtractQueryAnchors(text string) map[string]struct{}
- func FillRemainingSlots(keepIndices map[int]struct{}, items []any, n, effectiveMax int) map[int]struct{}
- func HashFieldName(fieldName string) string
- func IsUUIDFormat(value string) bool
- func ItemMatchesAnchors(item *orderedmap.OrderedMap, anchors map[string]struct{}) bool
- func PrioritizeIndices(cfg *Config, keepIndices map[int]struct{}, items []any, n int, ...) map[int]struct{}
- type AnchorConfig
- type AnchorPattern
- type AnchorSelector
- type ArrayAnalysis
- type ArrayType
- type CompressionPlan
- type CompressionStrategy
- type Config
- type Constraint
- type CrushArrayResult
- type CrushEvent
- type CrushResult
- type CrushabilityAnalysis
- type FieldStats
- type KeepErrorsConstraint
- type KeepStructuralOutliersConstraint
- type Observer
- type Scorer
- type SmartAnalyzer
- func (a *SmartAnalyzer) AnalyzeArray(items []*orderedmap.OrderedMap) ArrayAnalysis
- func (a *SmartAnalyzer) AnalyzeCrushability(items []*orderedmap.OrderedMap, fieldStats *orderedmap.OrderedMap) CrushabilityAnalysis
- func (a *SmartAnalyzer) AnalyzeField(key string, items []*orderedmap.OrderedMap) FieldStats
- func (a *SmartAnalyzer) DetectChangePoints(values []float64, window int) []int
- func (a *SmartAnalyzer) DetectPattern(fieldStats *orderedmap.OrderedMap, items []*orderedmap.OrderedMap) string
- func (a *SmartAnalyzer) DetectTemporalField(fieldStats *orderedmap.OrderedMap, items []*orderedmap.OrderedMap) bool
- func (a *SmartAnalyzer) EstimateReduction(fieldStats *orderedmap.OrderedMap, strategy CompressionStrategy, itemCount int) float64
- func (a *SmartAnalyzer) SelectStrategy(fieldStats *orderedmap.OrderedMap, pattern string, itemCount int, ...) CompressionStrategy
- type SmartCrusher
- func (c *SmartCrusher) CcrGet(hash string) (string, bool)
- func (c *SmartCrusher) CcrLen() int
- func (c *SmartCrusher) CompactDocumentJSON(doc string) (string, error)
- func (c *SmartCrusher) Crush(content, query string, bias float64) offloads.CrushResult
- func (c *SmartCrusher) CrushArrayJSON(arrayJSON, queryContext string, bias float64) (CrushArrayResult, error)
- type SmartCrusherBuilder
- func (b *SmartCrusherBuilder) AddConstraint(c Constraint) *SmartCrusherBuilder
- func (b *SmartCrusherBuilder) AddDefaultOSSConstraints() *SmartCrusherBuilder
- func (b *SmartCrusherBuilder) AddObserver(o Observer) *SmartCrusherBuilder
- func (b *SmartCrusherBuilder) AnchorConfig(cfg AnchorConfig) *SmartCrusherBuilder
- func (b *SmartCrusherBuilder) Build() *SmartCrusher
- func (b *SmartCrusherBuilder) WithCCRStore(store ccr.Store) *SmartCrusherBuilder
- func (b *SmartCrusherBuilder) WithCompaction(stage compaction.CompactionStage) *SmartCrusherBuilder
- func (b *SmartCrusherBuilder) WithDefaultCCRStore() *SmartCrusherBuilder
- func (b *SmartCrusherBuilder) WithDefaultCompaction() *SmartCrusherBuilder
- func (b *SmartCrusherBuilder) WithDefaultOSSSetup() *SmartCrusherBuilder
- func (b *SmartCrusherBuilder) WithScorer(scorer Scorer) *SmartCrusherBuilder
- type SmartCrusherPlanner
- type SummaryRange
- type TopValue
- type TracingObserver
Constants ¶
This section is empty.
Variables ¶
var ErrorKeywords = []string{
"error", "exception", "failed", "failure", "critical", "fatal",
"crash", "panic", "abort", "timeout", "denied", "rejected",
}
ErrorKeywords is the fallback error-preservation signal: an item whose lowercased JSON contains any of these substrings is kept. The list is lowercase by construction; the CALLER lowercases the haystack and does plain substring containment (not whole-word/prefix/regex), biasing toward over-preservation. The rustdoc inside constraints.rs wrongly lists 11 (omitting "failure"); the authoritative count is 12 [ref: error_keywords.rs].
Functions ¶
func CalculateStringEntropy ¶
CalculateStringEntropy returns the normalized Shannon entropy over RUNES (code points), in [0,1]. Strings shorter than two runes, or with a single distinct rune, return 0.0 (maxEntropy log2(1)=0 is guarded) [ref: statistics.rs algorithm]. Rune-level is correct here (a statistical feature, not byte math).
func DeduplicateIndicesByContent ¶
DeduplicateIndicesByContent collapses content-duplicate indices to the lowest index [ref: orchestration.rs deduplicate_indices_by_content]. Iterating the input set ascending, it records the first index seen for each content hash, skipping out-of-bounds indices. An empty input yields an empty set.
func DetectErrorItemsForPreservation ¶
func DetectErrorItemsForPreservation(items []*orderedmap.OrderedMap, itemStrings []string) []int
DetectErrorItemsForPreservation returns the ascending indices of object items whose lowercased JSON representation contains any ErrorKeywords substring [ref: outliers.rs detect_error_items_for_preservation]. When itemStrings is provided and long enough, the cached string for an index is used verbatim (lowercased); otherwise the item is freshly serialized. Non-object (nil) elements are skipped BEFORE the cache lookup.
func DetectIDFieldStatistically ¶
func DetectIDFieldStatistically(stats FieldStats, values []any) (bool, float64)
DetectIDFieldStatistically reports whether the field is an identifier column and, if so, a confidence in [0,0.95]. The UniqueRatio<0.9 hard gate short- circuits first; then string (UUID / high-entropy) and numeric (sequential / range) branches, then a >0.98-uniqueness catch-all [ref: field_detect.rs detect_id]. All uniqueness comparisons are strict.
func DetectRareStatusValues ¶
func DetectRareStatusValues(items []*orderedmap.OrderedMap, commonFields map[string]struct{}) []int
DetectRareStatusValues returns the indices of items whose value on a common field falls OUTSIDE that field's Pareto-dominant cluster [ref: outliers.rs detect_rare_status_values]. The result is APPEND-ordered (scan order) and may contain duplicate indices across fields; the caller (DetectStructuralOutliers) deduplicates. A field qualifies only when its distinct non-null values number between 2 and 50 inclusive and no more than 5 values are needed to reach 80% coverage.
func DetectScoreFieldStatistically ¶
func DetectScoreFieldStatistically(stats FieldStats, items []any) (bool, float64)
DetectScoreFieldStatistically reports whether the field is a score/relevance column and, if so, a confidence in [0,0.95]. It requires a numeric field with both min and max set, accumulates range/descending/float bonuses, and rejects sequential columns [ref: field_detect.rs detect_score]. The [-1,1] range check is intentionally ASYMMETRIC (min>=-1 AND max<=1) — do NOT symmetrize.
func DetectSequentialPattern ¶
DetectSequentialPattern reports whether values form a near-arithmetic sequence with a unit-ish step (avg diff in [0.5,2.0]). BUG#2 fixed: purely-string numerics are treated as categorical and rejected — at least one real JSON Number must be present [ref: statistics.rs algorithm, BUG#2]. When checkOrder is set, the ORIGINAL (unsorted) order must be mostly ascending too.
func DetectStructuralOutliers ¶
func DetectStructuralOutliers(items []*orderedmap.OrderedMap) []int
DetectStructuralOutliers returns the ascending-sorted, deduplicated indices of items that look structurally anomalous: items carrying a rare field, plus items carrying a rare status value on a common field [ref: outliers.rs detect_structural_outliers]. Non-object (nil) elements are skipped in field scans but count toward the denominator n, which shifts the common/rare thresholds. An array shorter than 5 yields no outliers.
func ExtractQueryAnchors ¶
ExtractQueryAnchors returns the lowercased anchor set for text. The set is a map[string]struct{} because order is irrelevant on the query side (this is not the compression path). Empty text yields an empty set. Each regex runs over the FULL original text independently, so a substring may contribute to multiple categories [ref: anchors.rs extract_query_anchors].
func FillRemainingSlots ¶
func FillRemainingSlots(keepIndices map[int]struct{}, items []any, n, effectiveMax int) map[int]struct{}
FillRemainingSlots tops the keep set up toward effectiveMax with content-unique, stride-sampled candidates [ref: orchestration.rs fill_remaining_slots]. When the set is already at/over budget it is returned unchanged. Candidates are the in-bounds indices NOT already kept, walked in a stride pattern so the fill is spread across the array rather than clustered at the head.
func HashFieldName ¶
HashFieldName returns the first 8 lowercase-hex chars of the SHA-256 digest of the field name's UTF-8 bytes. It mirrors Python hashlib.sha256(name.encode()).hexdigest()[:8] and is used as a cache key to look up TOIN-anonymized preserve_fields (TOIN stores names as SHA-256[:8]).
This is NOT a CCR marker hash and never emits <<ccr:...>>. The truncation is load-bearing: it MUST be [:8] — a prior [:16] silently broke all preserve-field lookups. Hash the bytes, not the runes ("café" -> "850f7dc4") [ref: hashing.rs].
func IsUUIDFormat ¶
IsUUIDFormat reports whether value is a canonical 36-byte 8-4-4-4-12 hex UUID. Length is measured in BYTES (Rust .len()), hex is case-insensitive, and no version/variant bits are validated [ref: statistics.rs algorithm].
func ItemMatchesAnchors ¶
func ItemMatchesAnchors(item *orderedmap.OrderedMap, anchors map[string]struct{}) bool
ItemMatchesAnchors reports whether the Python-repr of item (lowercased) contains any anchor as a substring [ref: anchors.rs item_matches_anchors]. An empty anchor set never matches. pythonRepr includes single-quoted object keys, so matching tests BOTH keys and values.
func PrioritizeIndices ¶
func PrioritizeIndices(cfg *Config, keepIndices map[int]struct{}, items []any, n int, analysis *ArrayAnalysis, effectiveMax int) map[int]struct{}
PrioritizeIndices resolves the final keep set under a budget [ref: orchestration.rs prioritize_indices]. It dedups (when enabled), fills toward the budget, and returns early if the result is within budget. Over budget, it keeps ALL critical rows (errors, structural outliers, numeric anomalies) plus the first-3/last-2 anchors plus remaining fill — the result MAY exceed effectiveMax because criticals are never dropped. analysis may be nil.
Types ¶
type AnchorConfig ¶
AnchorConfig holds the head/tail anchor counts [ref: (d) SAMPLING first_anchors = 3, last_anchors = 2].
func DefaultAnchorConfig ¶
func DefaultAnchorConfig() AnchorConfig
DefaultAnchorConfig returns the MVP anchor counts: 3 leading, 2 trailing.
type AnchorPattern ¶
type AnchorPattern int
AnchorPattern names the array shape the selector is anchoring for. Only Generic is exercised in the MVP; the other variants exist so planning.go's mapToAnchorPattern (Task 13) has a stable target set.
const ( // AnchorGeneric is the default/fallback pattern (SmartSample -> Generic). AnchorGeneric AnchorPattern = iota // AnchorTimeSeries anchors time-ordered arrays. // // DEFERRED (Plan 4+): no distinct MVP anchoring; treated as Generic. AnchorTimeSeries // AnchorSearchResults anchors ranked search-result arrays (from TopN). // // DEFERRED (Plan 4+): no distinct MVP anchoring; treated as Generic. AnchorSearchResults // AnchorLogs anchors clustered log arrays (from ClusterSample). // // DEFERRED (Plan 4+): no distinct MVP anchoring; treated as Generic. AnchorLogs )
type AnchorSelector ¶
type AnchorSelector struct {
// contains filtered or unexported fields
}
AnchorSelector selects anchor indices per its AnchorConfig.
func NewAnchorSelector ¶
func NewAnchorSelector(cfg AnchorConfig) *AnchorSelector
NewAnchorSelector builds a selector from cfg.
func (*AnchorSelector) SelectAnchors ¶
func (s *AnchorSelector) SelectAnchors(items []*orderedmap.OrderedMap, maxItems int, pattern AnchorPattern, query string) []int
SelectAnchors returns the ascending, deduplicated, in-bounds set of anchor indices for items: the first min(FirstAnchors, n), the last LastAnchors, and every index whose item matches a query anchor. maxItems and pattern are accepted for signature parity with the DEFERRED per-pattern selectors but do not affect the MVP Generic selection. An empty array yields an empty set.
type ArrayAnalysis ¶
type ArrayAnalysis struct {
ItemCount int
FieldStats *orderedmap.OrderedMap
DetectedPattern string
RecommendedStrategy CompressionStrategy
ConstantFields *orderedmap.OrderedMap
EstimatedReduction float64
Crushability *CrushabilityAnalysis
}
ArrayAnalysis is the full analysis of one array. FieldStats/ConstantFields are ordered maps (key order load-bearing). DetectedPattern domain: time_series|logs|search_results|generic.
type ArrayType ¶
type ArrayType int
ArrayType is the coarse element-homogeneity label ClassifyArray assigns to a JSON array. It is a pure value classifier — no CCR markers, no sampling. Only DictArray drives an MVP strategy (the lossless compaction-table path); the other labels' downstream crushers are DEFERRED (Plan 4+), but every label is emitted so routing sees the true shape.
func ClassifyArray ¶
ClassifyArray labels an array by its element types. It walks EVERY element (no sampling, no early break) and sets six presence flags, then applies the homogeneity decision. A single non-conforming element (including one null among objects, or a bool mixed with numbers) collapses the array to MixedArray. There is no pure-null gate, so [null, null] -> MixedArray.
Elements are expected in the shapes decodeJSON produces: *orderedmap.OrderedMap for objects, []any for arrays, json.Number for numbers, and string/bool/nil for the remaining scalars. The Go type switch separates bool from number for us (like Rust serde), so [true, false, 1] falls through to MixedArray without any special-casing.
type CompressionPlan ¶
type CompressionPlan struct {
Strategy CompressionStrategy
KeepIndices []int
ConstantFields *orderedmap.OrderedMap
SummaryRanges []SummaryRange
ClusterField *string
SortField *string
KeepCount int
}
CompressionPlan describes how to compress one array. ClusterField/SortField are nil in the MVP; SummaryRanges is empty.
func DefaultCompressionPlan ¶
func DefaultCompressionPlan() CompressionPlan
DefaultCompressionPlan returns a None-strategy plan with empty indices/ranges, nil cluster/sort fields, and the parity-pinned KeepCount of 10.
type CompressionStrategy ¶
type CompressionStrategy int
CompressionStrategy enumerates the recommended crush strategies. In the MVP only None/Skip/SmartSample are actively assigned; TimeSeries/ClusterSample/TopN are defined-but-unassigned parity variants.
const ( StrategyNone CompressionStrategy = iota StrategySkip StrategyTimeSeries StrategyClusterSample StrategyTopN StrategySmartSample )
func (CompressionStrategy) String ¶
func (s CompressionStrategy) String() string
String returns the parity-pinned lowercase identifier (byte-pinned; Decision B does NOT relax these). Note StrategyClusterSample is "cluster", NOT "cluster_sample".
type Config ¶
type Config struct {
Enabled bool
MinItemsToAnalyze int
MinTokensToCrush int
VarianceThreshold float64
UniquenessThreshold float64
SimilarityThreshold float64
MaxItemsAfterCrush int
PreserveChangePoints bool
FactorOutConstants bool
IncludeSummaries bool
UseFeedbackHints bool
ToinConfidenceThreshold float64
DedupIdenticalItems bool
FirstFraction float64
LastFraction float64
RelevanceThreshold float64
// LosslessMinSavingsRatio is the savings gate used in crushArray:
// savings = 1 - len(rendered)/len(input) (BYTES); savings < this ratio falls
// through to the lossy path. Upstream diverges: config.rs lowered the default
// to 0.15, but crusher.rs docs, the parity fixtures, and this plan's brief use
// 0.30. The MVP-effective default is 0.30; tests override to 0.99 to force the
// lossy path.
LosslessMinSavingsRatio float64
EnableCCRMarker bool
CompactionCoreFieldFraction float64
CompactionHeterogeneousCoreRatio float64
CompactionMaxFlattenInnerKeys int
CompactionMinBuckets int
CompactionMaxBuckets int
}
Config holds the SmartCrusher tuning knobs. It is a schema-preserving compressor: output contains only items from the original array. Fields tagged DEFERRED-consumer below are defined for parity but not yet exercised in the MVP.
func DefaultConfig ¶
func DefaultConfig() Config
DefaultConfig returns the canonical SmartCrusher defaults [ref: (d) constants].
type Constraint ¶
type Constraint interface {
Name() string
MustKeep(items []*orderedmap.OrderedMap, itemStrings []string) []int
}
Constraint yields the in-bounds item indices that MUST be kept. The allocator takes the union across the registered constraint stack. itemStrings is an optional pre-serialized JSON cache (nil when absent); it is a perf hint only and never changes the result. Implementations are pure and O(n) and must never return an out-of-bounds index (the allocator would silently drop it anyway).
type CrushArrayResult ¶
type CrushArrayResult struct {
Items []any
StrategyInfo string
CcrHash *string
DroppedSummary string
Compacted *string
CompactionKind *string
}
CrushArrayResult is the internal result of crushArray [ref: crusher.rs CrushArrayResult]. Items is the (possibly reduced) kept slice; Compacted, when non-nil, is the rendered lossless-table string that REPLACES the array wholesale; CcrHash/DroppedSummary describe a lossy row drop (space-form marker + store hash).
type CrushEvent ¶
type CrushEvent struct {
Strategy string
InputBytes int
OutputBytes int
ElapsedNs uint64
WasModified bool
}
CrushEvent is the per-Crush telemetry record consumed by Observers. All byte fields are BYTE lengths (Go len), NOT rune counts [ref: traits.rs].
DEFERRED (Plan 4+): only TracingObserver consumes this today; the seam carries no observer wiring, so the compression path never populates or emits it.
type CrushResult ¶
CrushResult is the INTERNAL 4-field result the crusher computes. It is RICHER than the offloads.CrushResult seam type (2 fields) and stays internal — do NOT widen the seam. Conflating the two is the #1 porting error.
func PassthroughResult ¶
func PassthroughResult(content string) CrushResult
PassthroughResult builds a no-op CrushResult (Compressed==Original==content, WasModified false, Strategy "passthrough").
type CrushabilityAnalysis ¶
type CrushabilityAnalysis struct {
Crushable bool
Confidence float64
Reason string
SignalsPresent []string
SignalsAbsent []string
HasIDField bool
IDUniqueness float64
AvgStringUniqueness float64
HasScoreField bool
ErrorItemCount int
AnomalyCount int
}
CrushabilityAnalysis is the analyzer's verdict for one array.
func SkipAnalysis ¶
func SkipAnalysis(reason string, confidence float64) CrushabilityAnalysis
SkipAnalysis builds a not-crushable CrushabilityAnalysis with metrics zeroed and slices EMPTY but non-nil.
type FieldStats ¶
type FieldStats struct {
Name string
FieldType string
Count int
UniqueCount int
UniqueRatio float64
IsConstant bool
ConstantValue *any
MinVal *float64
MaxVal *float64
MeanVal *float64
Variance *float64
ChangePoints []int
AvgLength *float64
TopValues []TopValue
}
FieldStats captures per-field statistics. Option<T> upstream fields map to Go pointers so absent (nil) and zero (&0.0) stay distinct. FieldType domain: numeric|string|boolean|object|array|null.
type KeepErrorsConstraint ¶
type KeepErrorsConstraint struct{}
KeepErrorsConstraint preserves items whose JSON contains an error keyword. It USES the itemStrings cache when provided (perf only; the result is identical to the nil path) [ref: constraints.rs keep_errors].
func (KeepErrorsConstraint) MustKeep ¶
func (KeepErrorsConstraint) MustKeep(items []*orderedmap.OrderedMap, itemStrings []string) []int
MustKeep delegates to DetectErrorItemsForPreservation, forwarding the optional itemStrings cache.
func (KeepErrorsConstraint) Name ¶
func (KeepErrorsConstraint) Name() string
Name reports the stable constraint identifier.
type KeepStructuralOutliersConstraint ¶
type KeepStructuralOutliersConstraint struct{}
KeepStructuralOutliersConstraint preserves items that look structurally anomalous (rare field or rare status value). It IGNORES itemStrings (upstream param is _item_strings) [ref: constraints.rs keep_structural_outliers].
func (KeepStructuralOutliersConstraint) MustKeep ¶
func (KeepStructuralOutliersConstraint) MustKeep(items []*orderedmap.OrderedMap, _ []string) []int
MustKeep delegates to DetectStructuralOutliers; the itemStrings argument is deliberately unused.
func (KeepStructuralOutliersConstraint) Name ¶
func (KeepStructuralOutliersConstraint) Name() string
Name reports the stable constraint identifier.
type Observer ¶
type Observer interface {
Name() string
OnEvent(event *CrushEvent)
}
Observer receives one CrushEvent per top-level Crush, synchronously, after the result is computed and before it is returned. Observers fire in registration order and MUST NOT panic (catch any I/O error internally).
DEFERRED (Plan 4+): telemetry only; the MVP seam does not plumb observers.
type Scorer ¶
Scorer is the relevance scoring contract. It is an ALIAS of relevance.Scorer, not a redefinition — do NOT define a second Scorer interface [ref: traits.rs Scorer re-export].
type SmartAnalyzer ¶
type SmartAnalyzer struct {
// contains filtered or unexported fields
}
SmartAnalyzer is a stateless verdict engine carrying only its config.
func NewSmartAnalyzer ¶
func NewSmartAnalyzer(config Config) *SmartAnalyzer
NewSmartAnalyzer builds an analyzer bound to config. Only VarianceThreshold and MinItemsToAnalyze are consulted here.
func (*SmartAnalyzer) AnalyzeArray ¶
func (a *SmartAnalyzer) AnalyzeArray(items []*orderedmap.OrderedMap) ArrayAnalysis
AnalyzeArray is the top entry point [ref: analyzer.rs AnalyzeArray]. It guards empty/first-not-object arrays into a None-strategy, nil-crushability result (generic pattern), otherwise builds the sorted-key FieldStats map, detects the pattern, factors constant fields, runs the crushability tree, selects a strategy, and estimates the reduction.
func (*SmartAnalyzer) AnalyzeCrushability ¶
func (a *SmartAnalyzer) AnalyzeCrushability(items []*orderedmap.OrderedMap, fieldStats *orderedmap.OrderedMap) CrushabilityAnalysis
AnalyzeCrushability is the six-case decision tree [ref: analyzer.rs AnalyzeCrushability]. Field iteration is ASCII-sorted throughout; case order is load-bearing (CASE0 precedes CASE1). It gathers signals (id/score/outliers/ error/anomalies/change_points), computes uniqueness aggregates, then returns the first matching verdict.
func (*SmartAnalyzer) AnalyzeField ¶
func (a *SmartAnalyzer) AnalyzeField(key string, items []*orderedmap.OrderedMap) FieldStats
AnalyzeField computes per-field statistics [ref: analyzer.rs AnalyzeField]. Missing keys and explicit nulls both count as null values.
func (*SmartAnalyzer) DetectChangePoints ¶
func (a *SmartAnalyzer) DetectChangePoints(values []float64, window int) []int
DetectChangePoints returns the deduped, ascending indices where the local mean shifts by more than variance_threshold*stdev [ref: analyzer.rs DetectChangePoints]. Series shorter than window*2, or with non-positive stdev, yield no change points. Subtraction i-window is saturating.
func (*SmartAnalyzer) DetectPattern ¶
func (a *SmartAnalyzer) DetectPattern(fieldStats *orderedmap.OrderedMap, items []*orderedmap.OrderedMap) string
DetectPattern classifies the array as time_series | logs | search_results | generic [ref: analyzer.rs DetectPattern]. Field iteration is ASCII-sorted.
func (*SmartAnalyzer) DetectTemporalField ¶
func (a *SmartAnalyzer) DetectTemporalField(fieldStats *orderedmap.OrderedMap, items []*orderedmap.OrderedMap) bool
DetectTemporalField reports whether any field looks like a timestamp [ref: analyzer.rs DetectTemporalField]. String fields sample the first 10 items and check the ISO-datetime/date fraction over the SAMPLED string count (not the full array length); numeric fields check ONLY min_val against the unix-seconds/millis ranges (max_val must merely be present).
func (*SmartAnalyzer) EstimateReduction ¶
func (a *SmartAnalyzer) EstimateReduction(fieldStats *orderedmap.OrderedMap, strategy CompressionStrategy, itemCount int) float64
EstimateReduction estimates the fractional size reduction [ref: analyzer.rs EstimateReduction]. itemCount is unused (kept for signature parity). None strategy or empty field_stats yield 0.0 (the empty guard also prevents an upstream ZeroDivisionError).
func (*SmartAnalyzer) SelectStrategy ¶
func (a *SmartAnalyzer) SelectStrategy(fieldStats *orderedmap.OrderedMap, pattern string, itemCount int, crushability *CrushabilityAnalysis) CompressionStrategy
SelectStrategy maps the crushability verdict + pattern to a CompressionStrategy [ref: analyzer.rs SelectStrategy]. TimeSeries/ClusterSample/TopN are computed for parity even though their crushers are DEFERRED; downstream falls back.
type SmartCrusher ¶
type SmartCrusher struct {
// contains filtered or unexported fields
}
SmartCrusher is the assembled crusher [ref: crusher.rs from_parts]. The struct and its fromParts constructor live here (in builder.go) because Build() needs to materialize it; the crush behavior (the offloads.Crusher seam impl, the 3-tier funnel, processValue, and the two public NewSmartCrusher* constructors) is added in crusher.go (Task 14). The planner field is derived inside fromParts from the other collaborators.
func NewSmartCrusher ¶
func NewSmartCrusher(cfg Config) *SmartCrusher
NewSmartCrusher builds the lossless-first OSS default crusher: the two default constraints + tracing observer, the CSV-schema compaction stage (Tier-2), and an in-memory CCR store for the lossy Tier-3 offloads. The blank import in builder.go guarantees the in-memory backend is registered, so Build() cannot fail.
func NewSmartCrusherWithoutCompaction ¶
func NewSmartCrusherWithoutCompaction(cfg Config) *SmartCrusher
NewSmartCrusherWithoutCompaction builds a lossy-only crusher (no compaction stage), still with an in-memory CCR store for the space-form lossy offloads. It forces every crushable array past the lossless tier straight to the lossy funnel.
func (*SmartCrusher) CcrGet ¶
func (c *SmartCrusher) CcrGet(hash string) (string, bool)
CcrGet retrieves a stored CCR payload by hash (test/round-trip helper). It returns ("", false) when the store is nil or the hash is absent.
func (*SmartCrusher) CcrLen ¶
func (c *SmartCrusher) CcrLen() int
CcrLen reports the CCR store entry count (0 when the store is nil).
func (*SmartCrusher) CompactDocumentJSON ¶
func (c *SmartCrusher) CompactDocumentJSON(doc string) (string, error)
CompactDocumentJSON drives the DocumentCompactor walker path with the crusher's ccrStore, so opaque blobs it emits ARE store-written and round-trippable (BLAKE3 24-hex). It decodes doc, walks it, and re-serializes the result compact. Used by the parity round-trip fixtures for the walker opaque path.
func (*SmartCrusher) Crush ¶
func (c *SmartCrusher) Crush(content, query string, bias float64) offloads.CrushResult
Crush is the offloads.Crusher seam entry point [ref: crusher.rs Crush]. It runs the crush and maps the INTERNAL result onto the 2-field seam type, dropping the Strategy/Original fields (do NOT widen the seam).
func (*SmartCrusher) CrushArrayJSON ¶
func (c *SmartCrusher) CrushArrayJSON(arrayJSON, queryContext string, bias float64) (CrushArrayResult, error)
CrushArrayJSON is a test helper: decode a JSON array string, run crushArray, and return the raw CrushArrayResult.
type SmartCrusherBuilder ¶
type SmartCrusherBuilder struct {
// contains filtered or unexported fields
}
SmartCrusherBuilder is the fluent composition point for a SmartCrusher [ref: builder.rs]. It carries zero compression logic — it only assembles the config, scorer, constraint/observer stacks, optional compaction stage, and optional CCR store, then wires them into a SmartCrusher via from_parts on Build(). A fresh builder is EMPTY (no scorer/constraints/observers/compaction/store); Build() defaults only the scorer (to HybridScorer). Setters return the builder for chaining and COMPOSE (append), never replace.
func NewSmartCrusherBuilder ¶
func NewSmartCrusherBuilder(config Config) *SmartCrusherBuilder
NewSmartCrusherBuilder returns an EMPTY builder holding config by value.
func (*SmartCrusherBuilder) AddConstraint ¶
func (b *SmartCrusherBuilder) AddConstraint(c Constraint) *SmartCrusherBuilder
AddConstraint appends a must-keep constraint (order preserved for determinism).
func (*SmartCrusherBuilder) AddDefaultOSSConstraints ¶
func (b *SmartCrusherBuilder) AddDefaultOSSConstraints() *SmartCrusherBuilder
AddDefaultOSSConstraints appends the OSS-default constraint stack [KeepErrors, KeepStructuralOutliers] in that order (composes, never replaces).
func (*SmartCrusherBuilder) AddObserver ¶
func (b *SmartCrusherBuilder) AddObserver(o Observer) *SmartCrusherBuilder
AddObserver appends an observer (fires in registration order).
func (*SmartCrusherBuilder) AnchorConfig ¶
func (b *SmartCrusherBuilder) AnchorConfig(cfg AnchorConfig) *SmartCrusherBuilder
AnchorConfig sets the anchor head/tail counts (defaulted at Build() when unset).
func (*SmartCrusherBuilder) Build ¶
func (b *SmartCrusherBuilder) Build() *SmartCrusher
Build wires the assembled parts into a SmartCrusher [ref: builder.rs Build]. It returns NO error (builder.rs goPortNotes: defaults can't fail). It defaults the anchor config and scorer when unset, clones the config for the analyzer, and hands everything to from_parts.
func (*SmartCrusherBuilder) WithCCRStore ¶
func (b *SmartCrusherBuilder) WithCCRStore(store ccr.Store) *SmartCrusherBuilder
WithCCRStore sets an explicit CCR store.
func (*SmartCrusherBuilder) WithCompaction ¶
func (b *SmartCrusherBuilder) WithCompaction(stage compaction.CompactionStage) *SmartCrusherBuilder
WithCompaction sets the compaction stage (run before the lossy path).
func (*SmartCrusherBuilder) WithDefaultCCRStore ¶
func (b *SmartCrusherBuilder) WithDefaultCCRStore() *SmartCrusherBuilder
WithDefaultCCRStore sets an in-memory CCR store (1000 entries / 5-min TTL). The blank import above guarantees the in-memory backend is registered, so store construction cannot fail; mustInMemoryStore panics only on the structurally impossible error.
func (*SmartCrusherBuilder) WithDefaultCompaction ¶
func (b *SmartCrusherBuilder) WithDefaultCompaction() *SmartCrusherBuilder
WithDefaultCompaction sets the OSS-default CSV-schema compaction stage.
func (*SmartCrusherBuilder) WithDefaultOSSSetup ¶
func (b *SmartCrusherBuilder) WithDefaultOSSSetup() *SmartCrusherBuilder
WithDefaultOSSSetup wires the OSS defaults: HybridScorer, the two default constraints, and the TracingObserver. It equals SmartCrusher::new(config).
func (*SmartCrusherBuilder) WithScorer ¶
func (b *SmartCrusherBuilder) WithScorer(scorer Scorer) *SmartCrusherBuilder
WithScorer plugs in a relevance scorer (enterprise seam). Build() falls back to HybridScorer when this is unset.
type SmartCrusherPlanner ¶
type SmartCrusherPlanner struct {
Config Config
AnchorSelector *AnchorSelector
Scorer Scorer
Analyzer *SmartAnalyzer
Constraints []Constraint
}
SmartCrusherPlanner turns an ArrayAnalysis into a CompressionPlan whose keep_indices is a sorted list of original indices to retain [ref: planning.rs]. It emits NO CCR markers and does no byte rewriting (downstream concerns). The MVP covers the create_plan dispatcher (Skip short-circuit + SmartSample routing), plan_smart_sample, and the shared signal helpers; the heavy per-pattern planners (TopN/ClusterSample/TimeSeries) are DEFERRED and route to planSmartSample.
func NewSmartCrusherPlanner ¶
func NewSmartCrusherPlanner(config Config, anchorSelector *AnchorSelector, scorer Scorer, analyzer *SmartAnalyzer, constraints []Constraint) *SmartCrusherPlanner
NewSmartCrusherPlanner assembles a planner from its collaborators.
func (*SmartCrusherPlanner) CreatePlan ¶
func (p *SmartCrusherPlanner) CreatePlan(analysis *ArrayAnalysis, items []*orderedmap.OrderedMap, queryContext string, preserveFields []string, effectiveMaxItems *int, itemStrings []string) *CompressionPlan
CreatePlan dispatches on analysis.RecommendedStrategy [ref: planning.rs create_plan]. effectiveMaxItems overrides config.MaxItemsAfterCrush when non-nil. A Skip verdict short-circuits to keep-all (0..n) before any dispatch. All non-Skip strategies route through planSmartSample in the MVP; the heavy planners are DEFERRED. preserveFields and itemStrings are TOIN/perf hints (callers pass nil).
type SummaryRange ¶
SummaryRange is a summarized index span. Summary is *any-typed to distinguish absent from a zero value; SummaryRanges is empty in the MVP.
type TracingObserver ¶
type TracingObserver struct{}
TracingObserver is the OSS-default Observer: a stateless debug-log emitter [ref: observer.rs].
DEFERRED (Plan 4+): observability only, MVP-irrelevant to compression. The seam never plumbs observers, so OnEvent is not called on the compression path today; it is defined so WithDefaultOSSSetup can register the one default observer.
func (TracingObserver) Name ¶
func (TracingObserver) Name() string
Name reports the stable observer identifier.
func (TracingObserver) OnEvent ¶
func (TracingObserver) OnEvent(event *CrushEvent)
OnEvent debug-logs the crush event. It is zero-cost when debug logging is filtered (the level gate runs before any field evaluation) and MUST NOT panic — a nil event is tolerated.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package compaction implements the lossless compaction-table stage of the SmartCrusher: it turns a homogeneous JSON array-of-objects into a compact CSV-schema (or JSON/markdown-kv) rendering, replacing bulky opaque leaves with CCR cell markers.
|
Package compaction implements the lossless compaction-table stage of the SmartCrusher: it turns a homogeneous JSON array-of-objects into a compact CSV-schema (or JSON/markdown-kv) rendering, replacing bulky opaque leaves with CCR cell markers. |