routing

package
v1.261116.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: MPL-2.0 Imports: 19 Imported by: 0

Documentation

Index

Constants

View Source
const (
	SourceHealth       = "health"
	SourceAffinity     = "affinity"
	SourceSmartRouting = "smart_routing"
	SourceLoadBalancer = "load_balancer"
	// SourceProbePin marks the X-Tingly-Probe-Service bypass, which pins a
	// specific service without running the pipeline.
	SourceProbePin = "probe_pin"
)

Selection sources: which stage produced a SelectionResult. Also used as the stage names in EvaluatedStages and observability output.

Variables

This section is empty.

Functions

func AffinitySessionKey

func AffinitySessionKey(sessionID string, matchedSmartRuleIndex int) string

AffinitySessionKey returns the affinity-store session key scoped to the content partition smart routing matched: the bare session for the rule's top-level pool (no match, index -1), or session + "#sr<idx>" inside a smart subset. Partition-scoping is what lets one session hold independent pins per request kind (e.g. a Claude Code main pin and a subagent pin), each with its own prompt-cache continuity.

The partition identity is the smart rule's index, the only stable handle the config offers. Editing the rule list can renumber partitions and mis-bucket existing pins; pins are in-memory with short TTLs, so this self-heals within one TTL.

func ContainsService

func ContainsService(services []*loadbalance.Service, target *loadbalance.Service) bool

ContainsService checks if target exists in services by service ID.

func FilterActiveServices

func FilterActiveServices(services []*loadbalance.Service) []*loadbalance.Service

FilterActiveServices returns only active services from the input list

func IntersectServices

func IntersectServices(left, right []*loadbalance.Service) []*loadbalance.Service

IntersectServices keeps services that are present in both lists.

func OpenAIRequestForTest added in v0.260813.1

func OpenAIRequestForTest(model string) *openai.ChatCompletionNewParams

func ResolveSessionID

func ResolveSessionID(c *gin.Context, req interface{}) typ.SessionID

ResolveSessionID returns the best available session identifier from the request. Priority: X-Tingly-Session-ID header > native client header > Anthropic metadata.user_id > ClientIP. The client IP is always stored in IPBackup as a fallback for rate limiting or logging.

func ServiceForTest added in v0.260813.1

func ServiceForTest(provider, model string, active bool) *loadbalance.Service

ServiceForTest and OpenAIRequestForTest are exported wrappers around this package's own fixtures, for external test packages (e.g. routing_test's benchmarks, which must live outside package routing to import protocolserver without an import cycle) that need identical shapes without reimplementing them.

func TestFixtures_helpers

func TestFixtures_helpers(t *testing.T)

Types

type AffinityEntry

type AffinityEntry struct {
	Service   *loadbalance.Service
	MessageID string
	LockedAt  time.Time
	ExpiresAt time.Time
}

AffinityEntry represents a locked service for a session

type AffinityStage

type AffinityStage struct {
	// contains filtered or unexported fields
}

AffinityStage checks if a session has a locked service from previous requests. If found and valid, returns the locked service; otherwise passes to next stage.

It runs AFTER SmartRoutingStage, so pins are scoped to the content partition smart routing matched (see AffinitySessionKey): a session pinned inside one smart subset cannot drag requests that match a different subset — content routing decides the partition, affinity provides stickiness within it.

func NewAffinityStage

func NewAffinityStage(store AffinityStore) *AffinityStage

NewAffinityStage creates a new affinity stage backed by the given store

func (*AffinityStage) Evaluate

func (s *AffinityStage) Evaluate(ctx *SelectionContext, candidates []*loadbalance.Service) ([]*loadbalance.Service, *SelectionResult, error)

Evaluate checks for locked service affinity

func (*AffinityStage) Name

func (s *AffinityStage) Name() string

Name returns the stage identifier

type AffinityStore

type AffinityStore interface {
	Get(ruleUUID, sessionID string) (*AffinityEntry, bool)
	Set(ruleUUID, sessionID string, entry *AffinityEntry)
	CountByService(serviceID string) int // count active sessions locked to this service
}

AffinityStore interface defines operations for session-service affinity

type HealthFilter

type HealthFilter struct {
	// contains filtered or unexported fields
}

HealthFilter filters services based on their health status

func NewHealthFilter

func NewHealthFilter(monitor *loadbalance.HealthMonitor) *HealthFilter

NewHealthFilter creates a new health filter with the given health monitor

func (*HealthFilter) Filter

func (hf *HealthFilter) Filter(services []*loadbalance.Service) []*loadbalance.Service

Filter filters out unhealthy services from the given list Returns only healthy services. If no healthy services are found, returns an empty slice (not nil).

func (*HealthFilter) FilterWithFallback

func (hf *HealthFilter) FilterWithFallback(services []*loadbalance.Service) []*loadbalance.Service

FilterWithFallback filters services and returns healthy ones. If no healthy services are found, returns all services as a fallback. This is useful when you want to avoid complete failure even if all services are marked unhealthy.

func (*HealthFilter) GetHealthMonitor

func (hf *HealthFilter) GetHealthMonitor() *loadbalance.HealthMonitor

GetHealthMonitor returns the underlying health monitor

func (*HealthFilter) IsHealthy

func (hf *HealthFilter) IsHealthy(serviceID string) bool

IsHealthy checks if a specific service is healthy

type HealthStage

type HealthStage struct {
	// contains filtered or unexported fields
}

HealthStage filters unhealthy services from the context. It runs first and narrows ctx.CandidateServices.

func NewHealthStage

func NewHealthStage(filter *HealthFilter) *HealthStage

NewHealthStage creates a new health stage with the given health filter

func (*HealthStage) Evaluate

func (s *HealthStage) Evaluate(ctx *SelectionContext, candidates []*loadbalance.Service) ([]*loadbalance.Service, *SelectionResult, error)

func (*HealthStage) Name

func (s *HealthStage) Name() string

type LoadBalancer

type LoadBalancer interface {
	SelectService(rule *typ.Rule) (*loadbalance.Service, error)
}

LoadBalancer defines the interface for load balancing operations. This avoids importing the server package (which would create circular imports).

type LoadBalancerStage

type LoadBalancerStage struct {
	// contains filtered or unexported fields
}

LoadBalancerStage performs standard load balancing across all rule services. This stage always returns a service (or error), acting as the final fallback.

func NewLoadBalancerStage

func NewLoadBalancerStage(lb LoadBalancer) *LoadBalancerStage

NewLoadBalancerStage creates a new load balancer stage

func (*LoadBalancerStage) Evaluate

Evaluate selects a service using load balancing. This is the terminal stage: a failure here is a real error (there is no next stage to fall through to), so it is reported via err rather than a silent pass-through.

func (*LoadBalancerStage) Name

func (s *LoadBalancerStage) Name() string

Name returns the stage identifier

type ProviderResolver

type ProviderResolver interface {
	GetProviderByUUID(uuid string) (*typ.Provider, error)
	// GetEffectiveAffinity returns the effective affinity TTL for a rule,
	// considering both scenario default and rule override. Returns 0 if disabled.
	GetEffectiveAffinity(rule *typ.Rule) time.Duration
}

ProviderResolver resolves providers by UUID and persists config state.

type QuotaProvider added in v0.260827.1

type QuotaProvider interface {
	GetQuota(ctx context.Context, providerUUID string) (*quota.ProviderUsage, error)
}

QuotaProvider gives the smart-routing stage read access to quota usage (ai/quota) so the service_quota position can compare against provider usage. The stage only declares what it needs — usage for a provider — and leaves freshness policy to the implementation (quota.Manager serves stored data kept fresh by its background refresher). *quota.Manager satisfies this.

type SelectionContext

type SelectionContext struct {
	// Rule is the routing rule being evaluated
	Rule *typ.Rule

	// Request is the parsed API request (OpenAI/Anthropic params)
	Request interface{}

	// SessionID is the resolved session identifier for affinity
	// Priority: Tingly header > native client header > Anthropic metadata.user_id > ClientIP
	SessionID typ.SessionID

	// GinContext provides access to HTTP headers and client info
	GinContext *gin.Context

	// Scenario identifies the request type (openai, anthropic, etc.)
	Scenario typ.RuleScenario

	// MatchedSmartRuleIndex tracks which smart routing rule matched (-1 if none)
	// This is set by SmartRoutingStage and used for smart_rule-scoped affinity
	MatchedSmartRuleIndex int

	// BypassedSmartRules records smart-routing rules whose processors have
	// already run and bypassed the stage (returned (nil, false) to let the
	// pipeline continue). SmartRoutingStage skips these on re-evaluation so
	// processors do not loop on residual matchable content.
	BypassedSmartRules map[int]struct{}
}

SelectionContext encapsulates all input needed for service selection. It is created once per request and passed through the selection pipeline.

func NewSelectionContext

func NewSelectionContext(
	rule *typ.Rule,
	req interface{},
	c *gin.Context,
	scenario typ.RuleScenario,
) *SelectionContext

NewSelectionContext creates a new selection context with resolved session ID

type SelectionResult

type SelectionResult struct {
	// Service is the selected load-balanced service
	Service *loadbalance.Service

	// Provider is the resolved provider for the service
	Provider *typ.Provider

	// Source indicates which stage selected this service (one of the
	// Source* constants above).
	Source string

	// EvaluatedStages tracks which stages were evaluated (for observability)
	EvaluatedStages []string

	// MatchedSmartRuleIndex is the index of the matched smart routing rule
	// -1 if no smart routing rule matched
	MatchedSmartRuleIndex int
}

SelectionResult represents the output of service selection pipeline. It includes the selected service, provider, and metadata about the selection. A SelectionResult only ever represents a TERMINAL pick (a stage's `final` return value) — non-terminal narrowing is carried by the `narrowed` slice SelectionStage.Evaluate returns alongside it, not by this struct.

func NewResult

func NewResult(service *loadbalance.Service, source string) *SelectionResult

NewResult creates a new selection result with the given service and source

type SelectionStage

type SelectionStage interface {
	// Name returns the stage identifier for logging, metrics, and the
	// Source* constants on SelectionResult.
	Name() string

	// Evaluate narrows candidates (or returns them unchanged) and reports
	// whether this stage picked the final winner.
	//
	//   - err != nil: this stage could not be evaluated; the pipeline
	//     aborts and the error propagates to the caller of Select.
	//   - final != nil: this stage selected the winning service; the
	//     pipeline stops here. The returned narrowed slice is ignored in
	//     this case.
	//   - otherwise: narrowed becomes the input candidate set for the next
	//     stage (or, if this is the last stage, the pipeline reports "no
	//     service available").
	Evaluate(ctx *SelectionContext, candidates []*loadbalance.Service) (narrowed []*loadbalance.Service, final *SelectionResult, err error)
}

SelectionStage is one step in the service-selection pipeline. Stages are pure with respect to the pipeline's control flow: each receives the current candidate set as an explicit argument and returns what it asserts the set now is. There is no shared mutable state between stages, and no "nil means unchanged" — a stage with no opinion returns its input slice unchanged.

Stages may only READ fields on the *loadbalance.Service values in candidates; they must not write to them (Service.InitializeStats's idempotent lazy-init is the one sanctioned exception — see its doc comment). Narrowing must allocate a fresh slice — never truncate or append onto the input slice's backing array — so a caller's slice is never mutated out from under it.

type ServiceSelector

type ServiceSelector struct {
	// contains filtered or unexported fields
}

ServiceSelector is the main entry point for service selection. It orchestrates a pipeline of selection stages and validates the final result.

func NewServiceSelector

func NewServiceSelector(
	cfg ProviderResolver,
	affinity AffinityStore,
	lb LoadBalancer,
) *ServiceSelector

NewServiceSelector creates a new service selector

func NewServiceSelectorWithLogger

func NewServiceSelectorWithLogger(
	cfg ProviderResolver,
	affinity AffinityStore,
	lb LoadBalancer,
	multiLogger *pkgobs.MultiLogger,
) *ServiceSelector

NewServiceSelectorWithLogger is like NewServiceSelector but also wires the multi-logger into smart-routing stages so each evaluation is captured to the dedicated smart_routing log source.

func (*ServiceSelector) Select

Select is the main entry point for service selection. It picks a pre-built pipeline based on rule configuration and executes it.

func (*ServiceSelector) SetQuotaProvider added in v0.260827.1

func (s *ServiceSelector) SetQuotaProvider(qp QuotaProvider)

SetQuotaProvider wires quota lookups into the pipeline's smart routing stage so service_quota ops can compare against provider usage. Optional — call it after construction once a quota manager is available; unset leaves service_quota ops passing through (see QuotaProvider).

type SimpleSelector

type SimpleSelector struct {
	// contains filtered or unexported fields
}

SimpleSelector is the production-facing facade over the ServiceSelector pipeline. It adds the probe-pin bypass, gin session/observability plumbing, and debug headers on top of the raw stage pipeline. Request handlers should use it; only diagnostics (e.g. the LB simulator) drive ServiceSelector.Select directly.

func NewSimpleSelector

func NewSimpleSelector(selector *ServiceSelector) *SimpleSelector

NewSimpleSelector creates a simplified selector

func (*SimpleSelector) SelectService

func (s *SimpleSelector) SelectService(
	c *gin.Context,
	scenario typ.RuleScenario,
	rule *typ.Rule,
	req interface{},
) (*typ.Provider, *loadbalance.Service, error)

SelectService resolves the session, runs the selection pipeline, validates the provider, and records the decision (gin context keys, log line, debug headers). The resolved session ID is stored in the gin context.

func (*SimpleSelector) SelectServiceForEmbeddings

func (s *SimpleSelector) SelectServiceForEmbeddings(
	c *gin.Context,
	scenario typ.RuleScenario,
	rule *typ.Rule,
) (*typ.Provider, *loadbalance.Service, error)

SelectServiceForEmbeddings is a variant of SelectService for embedding requests. Embedding requests don't carry chat-style context, so content-based smart routing is skipped (load balancing, affinity, and health filters still apply).

func (*SimpleSelector) SelectServiceForImageGeneration

func (s *SimpleSelector) SelectServiceForImageGeneration(
	c *gin.Context,
	scenario typ.RuleScenario,
	rule *typ.Rule,
) (*typ.Provider, *loadbalance.Service, error)

SelectServiceForImageGeneration is a variant of SelectService for image generation requests. Image generation requests don't carry chat-style context, so content-based smart routing is skipped (load balancing, affinity, and health filters still apply).

type SmartRoutingStage

type SmartRoutingStage struct {
	// contains filtered or unexported fields
}

SmartRoutingStage evaluates smart routing rules and returns matched services. If multiple services match, applies load balancing within the matched set.

func NewSmartRoutingStage

func NewSmartRoutingStage(affinity AffinityStore) *SmartRoutingStage

NewSmartRoutingStage creates a new smart routing stage

func (*SmartRoutingStage) Evaluate

Evaluate evaluates smart routing rules and narrows the candidate set. It never terminates the pipeline (final is always nil) — it only decides WHICH scope the next stages see. When a matched rule carries op-level processors, the stage runs them (the "implicit bypass") so the LoadBalancer (global fallback) picks an upstream with the mutated request. The mutated request is NOT re-evaluated against the smart-routing rules — re-entering evaluation with a post-processor input invites surprising matches and is intentionally avoided.

func (*SmartRoutingStage) Name

func (s *SmartRoutingStage) Name() string

Name returns the stage identifier

func (*SmartRoutingStage) SetMultiLogger

func (s *SmartRoutingStage) SetMultiLogger(ml *pkgobs.MultiLogger)

SetMultiLogger attaches the multi-logger so the stage can emit structured smart-routing evaluation logs viewable from the frontend system log page.

func (*SmartRoutingStage) SetQuotaProvider added in v0.260827.1

func (s *SmartRoutingStage) SetQuotaProvider(qp QuotaProvider)

SetQuotaProvider wires quota lookups into the stage. Optional — when unset, service_quota ops see no data and pass through (see evaluateServiceQuotaOp in internal/routing/smartrouting/routing.go).

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址