Documentation
¶
Index ¶
- Constants
- func ComputeGuardrailsHash(rules []GuardrailRuleDescriptor) string
- func GuardrailsHashFromContext(ctx context.Context) string
- func IoReadAllBody(c *echo.Context) ([]byte, error)
- func ShouldSkipAllCache(req *http.Request) bool
- func ShouldSkipExactCache(req *http.Request) bool
- func WithGuardrailsHash(ctx context.Context, hash string) context.Context
- type GuardrailRuleDescriptor
- type InternalHandleResult
- type MapVecStore
- func (s *MapVecStore) Close() error
- func (s *MapVecStore) DeleteExpired(_ context.Context) error
- func (s *MapVecStore) Insert(_ context.Context, key string, vec []float32, response []byte, ...) error
- func (s *MapVecStore) Len() int
- func (s *MapVecStore) Search(_ context.Context, vec []float32, paramsHash string, limit int) ([]VecResult, error)
- type ResponseCacheMiddleware
- func (m *ResponseCacheMiddleware) Close() error
- func (m *ResponseCacheMiddleware) HandleInternalRequest(ctx context.Context, method, path string, body []byte, ...) (*InternalHandleResult, error)
- func (m *ResponseCacheMiddleware) HandleRequest(c *echo.Context, body []byte, next func() error) error
- func (m *ResponseCacheMiddleware) Middleware() echo.MiddlewareFunc
- type VecResult
- type VecStore
Constants ¶
const ( CacheTypeExact = "exact" CacheTypeSemantic = "semantic" CacheTypeBoth = "both" CacheHeaderExact = "HIT (exact)" CacheHeaderSemantic = "HIT (semantic)" )
CacheTypeHeader values for X-Cache-Type.
Variables ¶
This section is empty.
Functions ¶
func ComputeGuardrailsHash ¶
func ComputeGuardrailsHash(rules []GuardrailRuleDescriptor) string
ComputeGuardrailsHash computes the guardrails_hash for a set of rule identifiers. Each rule is represented as "name:type:order:mode:content_hash". The combined seed is sorted for stability, then passed through SHA-256. Uses xxhash64 per-component and SHA-256 for the final hash to balance speed and collision resistance.
func GuardrailsHashFromContext ¶
GuardrailsHashFromContext retrieves the guardrails hash from the context, using the core package's storage key.
func IoReadAllBody ¶
IoReadAllBody reads and restores c.Request().Body, returning the raw bytes. Safe to call multiple times — each call resets the body.
func ShouldSkipAllCache ¶
ShouldSkipAllCache reports whether caching must be bypassed for this request, matching the exact-cache middleware semantics for no-cache and no-store.
func ShouldSkipExactCache ¶
ShouldSkipExactCache reports whether the X-Cache-Type header requests semantic-only mode.
Types ¶
type GuardrailRuleDescriptor ¶
type GuardrailRuleDescriptor struct {
Name string
Type string
Order int
Mode string
Content string
}
GuardrailRuleDescriptor describes a single active guardrail rule for hashing.
type InternalHandleResult ¶
type InternalHandleResult struct {
StatusCode int
Headers http.Header
Body []byte
CacheType string
}
InternalHandleResult is the buffered result of running the cache middleware for a transport-free internal JSON request.
type MapVecStore ¶
type MapVecStore struct {
// contains filtered or unexported fields
}
MapVecStore is an in-memory VecStore for testing. It supports cosine-similarity search via dot-product on normalised vectors and TTL filtering.
func NewMapVecStore ¶
func NewMapVecStore() *MapVecStore
NewMapVecStore creates a new in-memory VecStore.
func (*MapVecStore) Close ¶
func (s *MapVecStore) Close() error
func (*MapVecStore) DeleteExpired ¶
func (s *MapVecStore) DeleteExpired(_ context.Context) error
func (*MapVecStore) Len ¶
func (s *MapVecStore) Len() int
type ResponseCacheMiddleware ¶
type ResponseCacheMiddleware struct {
// contains filtered or unexported fields
}
ResponseCacheMiddleware wraps response cache logic. App and server only see this type.
func NewResponseCacheMiddleware ¶
func NewResponseCacheMiddleware( cfg config.ResponseCacheConfig, resolvedProviders map[string]config.RawProviderConfig, usageLogger usage.LoggerInterface, pricingResolver usage.PricingResolver, ) (*ResponseCacheMiddleware, error)
NewResponseCacheMiddleware creates middleware from config. If neither simple nor semantic cache is configured, returns a no-op middleware. resolvedProviders must be the credential/env-resolved map from providers.InitResult (same keys as live routing). Semantic embedder.provider names a key in this map.
func NewResponseCacheMiddlewareWithStore ¶
func NewResponseCacheMiddlewareWithStore(store cache.Store, ttl time.Duration) *ResponseCacheMiddleware
NewResponseCacheMiddlewareWithStore creates middleware with a custom store (for testing).
func (*ResponseCacheMiddleware) Close ¶
func (m *ResponseCacheMiddleware) Close() error
Close waits for any in-flight cache writes to complete, then releases cache resources.
func (*ResponseCacheMiddleware) HandleInternalRequest ¶
func (m *ResponseCacheMiddleware) HandleInternalRequest( ctx context.Context, method, path string, body []byte, next func(*echo.Context) error, ) (*InternalHandleResult, error)
HandleInternalRequest runs the normal cache middleware for a transport-free internal request using a minimal synthetic HTTP/Echo context.
func (*ResponseCacheMiddleware) HandleRequest ¶
func (m *ResponseCacheMiddleware) HandleRequest(c *echo.Context, body []byte, next func() error) error
HandleRequest runs the full dual-layer cache check (exact then semantic) for a translated inference request that has already been guardrail-patched. body is the final patched request bytes; next is the real LLM call. Streaming and non-streaming requests are cached independently. Streaming misses persist raw SSE bytes and replay them verbatim on cache hits.
func (*ResponseCacheMiddleware) Middleware ¶
func (m *ResponseCacheMiddleware) Middleware() echo.MiddlewareFunc
Middleware returns the Echo middleware function for the exact-match (simple) cache. This is kept for backward compatibility but cache checks are now primarily done via Handle() inside the translated inference handlers, after guardrail patching.
type VecStore ¶
type VecStore interface {
// Search returns up to limit results whose params_hash matches and whose
// similarity score to vec is above the caller's threshold. Expired entries
// encountered during search are deleted before returning a miss.
Search(ctx context.Context, vec []float32, paramsHash string, limit int) ([]VecResult, error)
// Insert stores vec along with its response bytes and TTL metadata.
// expires_at is recorded as unix-seconds; Search and DeleteExpired filter on it.
Insert(ctx context.Context, key string, vec []float32, response []byte, paramsHash string, ttl time.Duration) error
// DeleteExpired removes all entries whose expires_at is in the past.
// Called periodically by a background goroutine started in NewVecStore.
DeleteExpired(ctx context.Context) error
Close() error
}
VecStore is the generic interface for all vector database backends. Each backend must implement similarity search, insertion with TTL metadata, and bulk expired-entry deletion for background cleanup.
func NewVecStore ¶
func NewVecStore(cfg config.VectorStoreConfig) (VecStore, error)
NewVecStore creates a VecStore for the backend selected by cfg.Type. Type must be non-empty and one of: qdrant, pgvector, pinecone, weaviate.
Source Files