Documentation
¶
Index ¶
- Constants
- func AffinitySessionKey(sessionID string, matchedSmartRuleIndex int) string
- func ContainsService(services []*loadbalance.Service, target *loadbalance.Service) bool
- func FilterActiveServices(services []*loadbalance.Service) []*loadbalance.Service
- func IntersectServices(left, right []*loadbalance.Service) []*loadbalance.Service
- func OpenAIRequestForTest(model string) *openai.ChatCompletionNewParams
- func ResolveSessionID(c *gin.Context, req interface{}) typ.SessionID
- func ServiceForTest(provider, model string, active bool) *loadbalance.Service
- func TestFixtures_helpers(t *testing.T)
- type AffinityEntry
- type AffinityStage
- type AffinityStore
- type HealthFilter
- func (hf *HealthFilter) Filter(services []*loadbalance.Service) []*loadbalance.Service
- func (hf *HealthFilter) FilterWithFallback(services []*loadbalance.Service) []*loadbalance.Service
- func (hf *HealthFilter) GetHealthMonitor() *loadbalance.HealthMonitor
- func (hf *HealthFilter) IsHealthy(serviceID string) bool
- type HealthStage
- type LoadBalancer
- type LoadBalancerStage
- type ProviderResolver
- type QuotaProvider
- type SelectionContext
- type SelectionResult
- type SelectionStage
- type ServiceSelector
- type SimpleSelector
- func (s *SimpleSelector) SelectService(c *gin.Context, scenario typ.RuleScenario, rule *typ.Rule, req interface{}) (*typ.Provider, *loadbalance.Service, error)
- func (s *SimpleSelector) SelectServiceForEmbeddings(c *gin.Context, scenario typ.RuleScenario, rule *typ.Rule) (*typ.Provider, *loadbalance.Service, error)
- func (s *SimpleSelector) SelectServiceForImageGeneration(c *gin.Context, scenario typ.RuleScenario, rule *typ.Rule) (*typ.Provider, *loadbalance.Service, error)
- type SmartRoutingStage
- func (s *SmartRoutingStage) Evaluate(ctx *SelectionContext, candidates []*loadbalance.Service) ([]*loadbalance.Service, *SelectionResult, error)
- func (s *SmartRoutingStage) Name() string
- func (s *SmartRoutingStage) SetMultiLogger(ml *pkgobs.MultiLogger)
- func (s *SmartRoutingStage) SetQuotaProvider(qp QuotaProvider)
Constants ¶
const ( SourceHealth = "health" SourceAffinity = "affinity" SourceSmartRouting = "smart_routing" SourceLoadBalancer = "load_balancer" // SourceProbePin marks the X-Tingly-Probe-Service bypass, which pins a // specific service without running the pipeline. SourceProbePin = "probe_pin" )
Selection sources: which stage produced a SelectionResult. Also used as the stage names in EvaluatedStages and observability output.
Variables ¶
This section is empty.
Functions ¶
func AffinitySessionKey ¶
AffinitySessionKey returns the affinity-store session key scoped to the content partition smart routing matched: the bare session for the rule's top-level pool (no match, index -1), or session + "#sr<idx>" inside a smart subset. Partition-scoping is what lets one session hold independent pins per request kind (e.g. a Claude Code main pin and a subagent pin), each with its own prompt-cache continuity.
The partition identity is the smart rule's index, the only stable handle the config offers. Editing the rule list can renumber partitions and mis-bucket existing pins; pins are in-memory with short TTLs, so this self-heals within one TTL.
func ContainsService ¶
func ContainsService(services []*loadbalance.Service, target *loadbalance.Service) bool
ContainsService checks if target exists in services by service ID.
func FilterActiveServices ¶
func FilterActiveServices(services []*loadbalance.Service) []*loadbalance.Service
FilterActiveServices returns only active services from the input list
func IntersectServices ¶
func IntersectServices(left, right []*loadbalance.Service) []*loadbalance.Service
IntersectServices keeps services that are present in both lists.
func OpenAIRequestForTest ¶ added in v0.260813.1
func OpenAIRequestForTest(model string) *openai.ChatCompletionNewParams
func ResolveSessionID ¶
ResolveSessionID returns the best available session identifier from the request. Priority: X-Tingly-Session-ID header > native client header > Anthropic metadata.user_id > ClientIP. The client IP is always stored in IPBackup as a fallback for rate limiting or logging.
func ServiceForTest ¶ added in v0.260813.1
func ServiceForTest(provider, model string, active bool) *loadbalance.Service
ServiceForTest and OpenAIRequestForTest are exported wrappers around this package's own fixtures, for external test packages (e.g. routing_test's benchmarks, which must live outside package routing to import protocolserver without an import cycle) that need identical shapes without reimplementing them.
func TestFixtures_helpers ¶
Types ¶
type AffinityEntry ¶
type AffinityEntry struct {
Service *loadbalance.Service
MessageID string
LockedAt time.Time
ExpiresAt time.Time
}
AffinityEntry represents a locked service for a session
type AffinityStage ¶
type AffinityStage struct {
// contains filtered or unexported fields
}
AffinityStage checks if a session has a locked service from previous requests. If found and valid, returns the locked service; otherwise passes to next stage.
It runs AFTER SmartRoutingStage, so pins are scoped to the content partition smart routing matched (see AffinitySessionKey): a session pinned inside one smart subset cannot drag requests that match a different subset — content routing decides the partition, affinity provides stickiness within it.
func NewAffinityStage ¶
func NewAffinityStage(store AffinityStore) *AffinityStage
NewAffinityStage creates a new affinity stage backed by the given store
func (*AffinityStage) Evaluate ¶
func (s *AffinityStage) Evaluate(ctx *SelectionContext, candidates []*loadbalance.Service) ([]*loadbalance.Service, *SelectionResult, error)
Evaluate checks for locked service affinity
func (*AffinityStage) Name ¶
func (s *AffinityStage) Name() string
Name returns the stage identifier
type AffinityStore ¶
type AffinityStore interface {
Get(ruleUUID, sessionID string) (*AffinityEntry, bool)
Set(ruleUUID, sessionID string, entry *AffinityEntry)
CountByService(serviceID string) int // count active sessions locked to this service
}
AffinityStore interface defines operations for session-service affinity
type HealthFilter ¶
type HealthFilter struct {
// contains filtered or unexported fields
}
HealthFilter filters services based on their health status
func NewHealthFilter ¶
func NewHealthFilter(monitor *loadbalance.HealthMonitor) *HealthFilter
NewHealthFilter creates a new health filter with the given health monitor
func (*HealthFilter) Filter ¶
func (hf *HealthFilter) Filter(services []*loadbalance.Service) []*loadbalance.Service
Filter filters out unhealthy services from the given list Returns only healthy services. If no healthy services are found, returns an empty slice (not nil).
func (*HealthFilter) FilterWithFallback ¶
func (hf *HealthFilter) FilterWithFallback(services []*loadbalance.Service) []*loadbalance.Service
FilterWithFallback filters services and returns healthy ones. If no healthy services are found, returns all services as a fallback. This is useful when you want to avoid complete failure even if all services are marked unhealthy.
func (*HealthFilter) GetHealthMonitor ¶
func (hf *HealthFilter) GetHealthMonitor() *loadbalance.HealthMonitor
GetHealthMonitor returns the underlying health monitor
func (*HealthFilter) IsHealthy ¶
func (hf *HealthFilter) IsHealthy(serviceID string) bool
IsHealthy checks if a specific service is healthy
type HealthStage ¶
type HealthStage struct {
// contains filtered or unexported fields
}
HealthStage filters unhealthy services from the context. It runs first and narrows ctx.CandidateServices.
func NewHealthStage ¶
func NewHealthStage(filter *HealthFilter) *HealthStage
NewHealthStage creates a new health stage with the given health filter
func (*HealthStage) Evaluate ¶
func (s *HealthStage) Evaluate(ctx *SelectionContext, candidates []*loadbalance.Service) ([]*loadbalance.Service, *SelectionResult, error)
func (*HealthStage) Name ¶
func (s *HealthStage) Name() string
type LoadBalancer ¶
type LoadBalancer interface {
SelectService(rule *typ.Rule) (*loadbalance.Service, error)
}
LoadBalancer defines the interface for load balancing operations. This avoids importing the server package (which would create circular imports).
type LoadBalancerStage ¶
type LoadBalancerStage struct {
// contains filtered or unexported fields
}
LoadBalancerStage performs standard load balancing across all rule services. This stage always returns a service (or error), acting as the final fallback.
func NewLoadBalancerStage ¶
func NewLoadBalancerStage(lb LoadBalancer) *LoadBalancerStage
NewLoadBalancerStage creates a new load balancer stage
func (*LoadBalancerStage) Evaluate ¶
func (s *LoadBalancerStage) Evaluate(ctx *SelectionContext, candidates []*loadbalance.Service) ([]*loadbalance.Service, *SelectionResult, error)
Evaluate selects a service using load balancing. This is the terminal stage: a failure here is a real error (there is no next stage to fall through to), so it is reported via err rather than a silent pass-through.
func (*LoadBalancerStage) Name ¶
func (s *LoadBalancerStage) Name() string
Name returns the stage identifier
type ProviderResolver ¶
type ProviderResolver interface {
GetProviderByUUID(uuid string) (*typ.Provider, error)
// GetEffectiveAffinity returns the effective affinity TTL for a rule,
// considering both scenario default and rule override. Returns 0 if disabled.
GetEffectiveAffinity(rule *typ.Rule) time.Duration
}
ProviderResolver resolves providers by UUID and persists config state.
type QuotaProvider ¶ added in v0.260827.1
type QuotaProvider interface {
GetQuota(ctx context.Context, providerUUID string) (*quota.ProviderUsage, error)
}
QuotaProvider gives the smart-routing stage read access to quota usage (ai/quota) so the service_quota position can compare against provider usage. The stage only declares what it needs — usage for a provider — and leaves freshness policy to the implementation (quota.Manager serves stored data kept fresh by its background refresher). *quota.Manager satisfies this.
type SelectionContext ¶
type SelectionContext struct {
// Rule is the routing rule being evaluated
Rule *typ.Rule
// Request is the parsed API request (OpenAI/Anthropic params)
Request interface{}
// SessionID is the resolved session identifier for affinity
// Priority: Tingly header > native client header > Anthropic metadata.user_id > ClientIP
SessionID typ.SessionID
// GinContext provides access to HTTP headers and client info
GinContext *gin.Context
// Scenario identifies the request type (openai, anthropic, etc.)
Scenario typ.RuleScenario
// MatchedSmartRuleIndex tracks which smart routing rule matched (-1 if none)
// This is set by SmartRoutingStage and used for smart_rule-scoped affinity
MatchedSmartRuleIndex int
// BypassedSmartRules records smart-routing rules whose processors have
// already run and bypassed the stage (returned (nil, false) to let the
// pipeline continue). SmartRoutingStage skips these on re-evaluation so
// processors do not loop on residual matchable content.
BypassedSmartRules map[int]struct{}
}
SelectionContext encapsulates all input needed for service selection. It is created once per request and passed through the selection pipeline.
func NewSelectionContext ¶
func NewSelectionContext( rule *typ.Rule, req interface{}, c *gin.Context, scenario typ.RuleScenario, ) *SelectionContext
NewSelectionContext creates a new selection context with resolved session ID
type SelectionResult ¶
type SelectionResult struct {
// Service is the selected load-balanced service
Service *loadbalance.Service
// Provider is the resolved provider for the service
Provider *typ.Provider
// Source indicates which stage selected this service (one of the
// Source* constants above).
Source string
// EvaluatedStages tracks which stages were evaluated (for observability)
EvaluatedStages []string
// MatchedSmartRuleIndex is the index of the matched smart routing rule
// -1 if no smart routing rule matched
MatchedSmartRuleIndex int
}
SelectionResult represents the output of service selection pipeline. It includes the selected service, provider, and metadata about the selection. A SelectionResult only ever represents a TERMINAL pick (a stage's `final` return value) — non-terminal narrowing is carried by the `narrowed` slice SelectionStage.Evaluate returns alongside it, not by this struct.
func NewResult ¶
func NewResult(service *loadbalance.Service, source string) *SelectionResult
NewResult creates a new selection result with the given service and source
type SelectionStage ¶
type SelectionStage interface {
// Name returns the stage identifier for logging, metrics, and the
// Source* constants on SelectionResult.
Name() string
// Evaluate narrows candidates (or returns them unchanged) and reports
// whether this stage picked the final winner.
//
// - err != nil: this stage could not be evaluated; the pipeline
// aborts and the error propagates to the caller of Select.
// - final != nil: this stage selected the winning service; the
// pipeline stops here. The returned narrowed slice is ignored in
// this case.
// - otherwise: narrowed becomes the input candidate set for the next
// stage (or, if this is the last stage, the pipeline reports "no
// service available").
Evaluate(ctx *SelectionContext, candidates []*loadbalance.Service) (narrowed []*loadbalance.Service, final *SelectionResult, err error)
}
SelectionStage is one step in the service-selection pipeline. Stages are pure with respect to the pipeline's control flow: each receives the current candidate set as an explicit argument and returns what it asserts the set now is. There is no shared mutable state between stages, and no "nil means unchanged" — a stage with no opinion returns its input slice unchanged.
Stages may only READ fields on the *loadbalance.Service values in candidates; they must not write to them (Service.InitializeStats's idempotent lazy-init is the one sanctioned exception — see its doc comment). Narrowing must allocate a fresh slice — never truncate or append onto the input slice's backing array — so a caller's slice is never mutated out from under it.
type ServiceSelector ¶
type ServiceSelector struct {
// contains filtered or unexported fields
}
ServiceSelector is the main entry point for service selection. It orchestrates a pipeline of selection stages and validates the final result.
func NewServiceSelector ¶
func NewServiceSelector( cfg ProviderResolver, affinity AffinityStore, lb LoadBalancer, ) *ServiceSelector
NewServiceSelector creates a new service selector
func NewServiceSelectorWithLogger ¶
func NewServiceSelectorWithLogger( cfg ProviderResolver, affinity AffinityStore, lb LoadBalancer, multiLogger *pkgobs.MultiLogger, ) *ServiceSelector
NewServiceSelectorWithLogger is like NewServiceSelector but also wires the multi-logger into smart-routing stages so each evaluation is captured to the dedicated smart_routing log source.
func (*ServiceSelector) Select ¶
func (s *ServiceSelector) Select(ctx *SelectionContext) (*SelectionResult, error)
Select is the main entry point for service selection. It picks a pre-built pipeline based on rule configuration and executes it.
func (*ServiceSelector) SetQuotaProvider ¶ added in v0.260827.1
func (s *ServiceSelector) SetQuotaProvider(qp QuotaProvider)
SetQuotaProvider wires quota lookups into the pipeline's smart routing stage so service_quota ops can compare against provider usage. Optional — call it after construction once a quota manager is available; unset leaves service_quota ops passing through (see QuotaProvider).
type SimpleSelector ¶
type SimpleSelector struct {
// contains filtered or unexported fields
}
SimpleSelector is the production-facing facade over the ServiceSelector pipeline. It adds the probe-pin bypass, gin session/observability plumbing, and debug headers on top of the raw stage pipeline. Request handlers should use it; only diagnostics (e.g. the LB simulator) drive ServiceSelector.Select directly.
func NewSimpleSelector ¶
func NewSimpleSelector(selector *ServiceSelector) *SimpleSelector
NewSimpleSelector creates a simplified selector
func (*SimpleSelector) SelectService ¶
func (s *SimpleSelector) SelectService( c *gin.Context, scenario typ.RuleScenario, rule *typ.Rule, req interface{}, ) (*typ.Provider, *loadbalance.Service, error)
SelectService resolves the session, runs the selection pipeline, validates the provider, and records the decision (gin context keys, log line, debug headers). The resolved session ID is stored in the gin context.
func (*SimpleSelector) SelectServiceForEmbeddings ¶
func (s *SimpleSelector) SelectServiceForEmbeddings( c *gin.Context, scenario typ.RuleScenario, rule *typ.Rule, ) (*typ.Provider, *loadbalance.Service, error)
SelectServiceForEmbeddings is a variant of SelectService for embedding requests. Embedding requests don't carry chat-style context, so content-based smart routing is skipped (load balancing, affinity, and health filters still apply).
func (*SimpleSelector) SelectServiceForImageGeneration ¶
func (s *SimpleSelector) SelectServiceForImageGeneration( c *gin.Context, scenario typ.RuleScenario, rule *typ.Rule, ) (*typ.Provider, *loadbalance.Service, error)
SelectServiceForImageGeneration is a variant of SelectService for image generation requests. Image generation requests don't carry chat-style context, so content-based smart routing is skipped (load balancing, affinity, and health filters still apply).
type SmartRoutingStage ¶
type SmartRoutingStage struct {
// contains filtered or unexported fields
}
SmartRoutingStage evaluates smart routing rules and returns matched services. If multiple services match, applies load balancing within the matched set.
func NewSmartRoutingStage ¶
func NewSmartRoutingStage(affinity AffinityStore) *SmartRoutingStage
NewSmartRoutingStage creates a new smart routing stage
func (*SmartRoutingStage) Evaluate ¶
func (s *SmartRoutingStage) Evaluate(ctx *SelectionContext, candidates []*loadbalance.Service) ([]*loadbalance.Service, *SelectionResult, error)
Evaluate evaluates smart routing rules and narrows the candidate set. It never terminates the pipeline (final is always nil) — it only decides WHICH scope the next stages see. When a matched rule carries op-level processors, the stage runs them (the "implicit bypass") so the LoadBalancer (global fallback) picks an upstream with the mutated request. The mutated request is NOT re-evaluated against the smart-routing rules — re-entering evaluation with a post-processor input invites surprising matches and is intentionally avoided.
func (*SmartRoutingStage) Name ¶
func (s *SmartRoutingStage) Name() string
Name returns the stage identifier
func (*SmartRoutingStage) SetMultiLogger ¶
func (s *SmartRoutingStage) SetMultiLogger(ml *pkgobs.MultiLogger)
SetMultiLogger attaches the multi-logger so the stage can emit structured smart-routing evaluation logs viewable from the frontend system log page.
func (*SmartRoutingStage) SetQuotaProvider ¶ added in v0.260827.1
func (s *SmartRoutingStage) SetQuotaProvider(qp QuotaProvider)
SetQuotaProvider wires quota lookups into the stage. Optional — when unset, service_quota ops see no data and pass through (see evaluateServiceQuotaOp in internal/routing/smartrouting/routing.go).