controller

package
v0.0.0-...-9c8e2d0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 11, 2026 License: AGPL-3.0 Imports: 52 Imported by: 0

Documentation

Index

Constants

View Source
const (
	ControlPlaneRootSecretName   = "clrk-control-plane-root"
	ControlPlaneServerSecretName = "clrk-control-plane-server"
	ControlPlaneCASecretName     = "clrk-control-plane-ca"
	ControlPlaneSecretLabel      = "clrk.apoxy.dev/control-plane-pki"
)

Control-plane PKI Secrets + label. All live in the controller-manager's runtime namespace (POD_NAMESPACE). The root issues the :9443 server cert and every per-EG client cert; the CA bundle is the mount-only trust anchor the data-plane Envoys reference to verify the server cert.

View Source
const (
	// EgressGatewayClassName is the shared GatewayClass we install for all
	// clrk EgressGateways. Envoy Gateway's Gateway controller binds Gateways
	// that reference this class to the clrk EnvoyProxy configuration.
	EgressGatewayClassName = "clrk-egress"

	// GatewayNamePrefix must match internal/egextension.GatewayNamePrefix.
	// Kept as a constant here to avoid importing the extension package into
	// the reconciler layer.
	GatewayNamePrefix = "clrk-eg-"

	// EgressGatewayFinalizer guards EgressGateway deletion so the
	// reconciler can reap the upstream-trust mirror Secret before the
	// object disappears. The mirror lives in the EG control-plane
	// namespace (EnvoyGatewayNamespace), not the EG's own namespace, so
	// it cannot carry an EG owner reference (SetControllerReference
	// rejects cross-namespace owners) and would otherwise leak on delete.
	EgressGatewayFinalizer = "clrk.apoxy.dev/egressgateway-upstream-trust"

	// EgressListenerBasePort is the first TCP port the EG-managed Envoy
	// binds. Each EgressListener in spec.listeners is assigned
	// EgressListenerBasePort+index — protocol shapes (TLS-Terminate,
	// plain TCP, etc.) cannot share a port, so each materializes as a
	// distinct Envoy listener and Service ServicePort.
	EgressListenerBasePort int32 = 18080

	// MaxEgressListeners caps spec.listeners to keep the assigned port
	// range narrow and reviewable. Bump if a real workload needs more.
	MaxEgressListeners = 8

	// LLMRevAnnotation carries a hash of the LLM routing inputs
	// (attached AIProviderRoutes, their referenced Backends, attached
	// FallbackRoutingPolicies) on the clrk-owned Gateway, for operator
	// observability. NOTE: the annotation alone does NOT trigger Envoy
	// Gateway retranslation — EG's watch predicate passes annotation
	// changes, but its gateway-api runner dedupes on the translated IR
	// and Gateway metadata never reaches the IR, so the xds-translator
	// (where the egextension hooks live) is never re-run. The actual
	// trigger is the revision header-match rule ensureCatchAllRoute
	// stamps on the catch-all HTTPRoute (see LLMRevHeaderName): route
	// rules ARE IR-visible, and the egextension replaces EG's whole
	// route configuration so the rule never serves traffic.
	LLMRevAnnotation = "clrk.apoxy.dev/llm-rev"

	// LLMRevHeaderName is the header name of the IR-perturbing revision
	// match on the catch-all HTTPRoute. Never sent by clients; exists
	// solely so a clrk CR change makes EG's translated IR differ and
	// forces the xds-translation pass that re-runs the extension hooks.
	LLMRevHeaderName = "x-clrk-llm-rev"
)
View Source
const (
	DispatchPort              = ports.DispatchPort
	IngressExtProcBackendName = ports.IngressExtProcBackendName
)

Re-exports of port + name consts so existing controller-package callers don't have to rewrite import paths. See internal/ports for the source of truth — that leaf package has no transitive deps, which is what lets cmd/clrk reach for them without pulling in generated protobuf code.

View Source
const ControlPlaneClientSecretLabel = "clrk.apoxy.dev/egressgateway-cp-client"

ControlPlaneClientSecretLabel marks a Secret as a per-EG control-plane mTLS client cert.

View Source
const EgressGatewayCASecretLabel = "clrk.apoxy.dev/egressgateway"

EgressGatewayCASecretLabel marks a Secret as the MITM CA for an EgressGateway.

Variables

This section is empty.

Functions

func ControlPlaneClientSecretName

func ControlPlaneClientSecretName(egNamespace, egName string) string

ControlPlaneClientSecretName is the deterministic name for a per-EG mTLS client-cert Secret. Like the upstream-trust mirror it lives in the EG control-plane namespace (r.EnvoyGatewayNamespace) so the EG-managed Envoy pod, which runs there, can mount it. Every EG across every namespace shares that one namespace, so the name embeds a short hash of the EG's namespaced name: two EGs of the same name in different namespaces must not collide onto a single Secret, or the second would mount the first's cert and present the wrong tenant identity (a namespace-qualified SAN) to the control plane.

func EgressGatewayCASecretName

func EgressGatewayCASecretName(egName string) string

EgressGatewayCASecretName returns the deterministic Secret name holding the MITM CA keypair for the named EgressGateway. The Secret lives in the EgressGateway's namespace. The private key is consumed by the MITM handshaker in controller-manager; sandbox workers only need the certificate PEM.

func EnsureControlPlaneCABundle

func EnsureControlPlaneCABundle(ctx context.Context, r client.Reader, w client.Client, ns string, caBundlePEM []byte) error

EnsureControlPlaneCABundle get-or-creates the shared CA-bundle Secret in ns that every data-plane Envoy mounts to verify the :9443 server cert. It holds only the root cert (no key) and is updated in place if the bundle changed (root rotation, rare).

func EnsureControlPlaneRoot

func EnsureControlPlaneRoot(ctx context.Context, r client.Reader, w client.Client, ns string) (rootCertPEM, rootKeyPEM []byte, err error)

EnsureControlPlaneRoot get-or-creates the control-plane root CA Secret in ns and returns its cert + key PEM. Reads go through r (an uncached reader -- callers run this at startup, before the manager cache is started); writes go through w. A create race (another replica or a restart) is resolved by re-reading the winner's Secret so every replica trusts a single shared root.

func EnsureControlPlaneServerCert

func EnsureControlPlaneServerCert(ctx context.Context, r client.Reader, w client.Client, ns, advertiseURI string, rootCertPEM, rootKeyPEM []byte) (serverCertPEM, serverKeyPEM, caBundlePEM []byte, err error)

EnsureControlPlaneServerCert get-or-creates the :9443 server-cert Secret in ns, signed by the given root, and returns the server cert/key PEM plus the CA bundle (the root cert). Desired SANs are derived from advertiseURI's host plus loopback names; an existing cert whose SANs no longer cover the desired set (advertiseURI changed across upgrades) is re-minted in place.

func MintEgressGatewayCA

func MintEgressGatewayCA(namespace, name string) (certPEM, keyPEM []byte, err error)

MintEgressGatewayCA generates a fresh ECDSA P-256 self-signed CA whose subject carries the EgressGateway coordinates for operator debuggability. It delegates to egpki.MintRootCA so the MITM CA and the control-plane PKI share one crypto shape (ECDSA P-256, 128-bit serial, SEC1 "EC PRIVATE KEY" PEM, 10-year validity); the only difference is the subject CommonName.

Types

type AIProviderRouteStatusReconciler

type AIProviderRouteStatusReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

AIProviderRouteStatusReconciler writes the previously-unwritten AIProviderRoute Status.Parents: an Accepted condition per clrk EgressGateway parentRef (does the gateway exist?) and a ResolvedRefs condition (do all clrk Backend backendRefs resolve?). It is status-only — it never mutates spec or the data plane — so Backend/BackendSelector references become discoverable and a typo'd ref surfaces as ResolvedRefs=False instead of silently degrading to pass-through.

func (*AIProviderRouteStatusReconciler) Reconcile

func (*AIProviderRouteStatusReconciler) SetupWithManager

func (r *AIProviderRouteStatusReconciler) SetupWithManager(mgr ctrl.Manager) error

type CLRKConfigReconciler

type CLRKConfigReconciler struct {
	client.Client
	Scheme *runtime.Scheme

	// Phone is the api.apoxy.dev client; nil when --phone-home=false, which
	// severs all outbound calls and reports Registered=False/PhoneHomeDisabled.
	Phone *sentry.Client

	// Namespace is the clrk system namespace where the CLRKConfig singleton and
	// the token Secret live.
	Namespace string

	// SecretName names the token Secret written on registration.
	SecretName string

	// ClrkVersion is reported to api.apoxy.dev at registration.
	ClrkVersion string

	// Dropped, when set, returns the cumulative count of security reports the
	// phone-home reporter has dropped under backpressure; it is stamped into
	// status.notifications.reportsDropped each reconcile so the console health
	// panel reflects real loss. Nil when no reporter is running (leader-only).
	Dropped func() int64
}

CLRKConfigReconciler drives the notifications phone-home registration for the CLRKConfig singleton. On signup (spec.notifications.email set) it registers the deployment with api.apoxy.dev, persists the returned bearer token to a core/v1 Secret (via the cluster client -- the embedded apiserver serves no core/v1), and records status.notifications (deploymentID + registrationTokenSecretRef + Registered condition). The browser sees only the Secret REFERENCE, never the token.

It also exposes AuthState so the sentry Reporter/AdvisoryPoller can resolve the live token + gating flags without a second CLRKConfig/Secret client.

func (*CLRKConfigReconciler) AuthState

AuthState resolves the current phone-home authorization by reading the CLRKConfig singleton and the token Secret. It is the AuthFunc for the sentry Reporter/AdvisoryPoller. A missing singleton or unset token yields an unauthorized (zero) state, not an error.

func (*CLRKConfigReconciler) Reconcile

func (r *CLRKConfigReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

func (*CLRKConfigReconciler) SetupWithManager

func (r *CLRKConfigReconciler) SetupWithManager(mgr ctrl.Manager) error

type CredentialInjectionPolicyStatusReconciler

type CredentialInjectionPolicyStatusReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

CredentialInjectionPolicyStatusReconciler writes the GEP-2649 PolicyStatus of a CredentialInjectionPolicy: one ancestor per targetRef with an Accepted condition reporting whether the (same-namespace) EgressGateway / AIProviderRoute / MCPRoute target exists. It is status-only — the worker resolves and applies the credential off the policy spec — so a typo'd or dangling targetRef surfaces as Accepted=False/TargetNotFound instead of silently never injecting.

func (*CredentialInjectionPolicyStatusReconciler) Reconcile

func (*CredentialInjectionPolicyStatusReconciler) SetupWithManager

type DaemonAgentRevisionReconciler

type DaemonAgentRevisionReconciler struct {
	client.Client
	Scheme *runtime.Scheme

	// Recorder emits rollout notifications (events.k8s.io/v1) on the
	// RevisionReady False->True crossing. Nil-safe.
	Recorder *notify.Recorder
}

DaemonAgentRevisionReconciler manages AgentSandboxRevision snapshots for a DaemonAgent. Phase / UpSince / RestartCount are written by the worker daemon loop — only the worker knows when its sandbox is alive.

func (*DaemonAgentRevisionReconciler) Reconcile

func (*DaemonAgentRevisionReconciler) SetupWithManager

func (r *DaemonAgentRevisionReconciler) SetupWithManager(mgr ctrl.Manager) error

type EgressGatewayOTLPForwardReconciler

type EgressGatewayOTLPForwardReconciler struct {
	client.Client

	// Registry is the per-EG forwarder map the receiver reads from.
	Registry *otelforward.Registry
}

EgressGatewayOTLPForwardReconciler watches EgressGateway objects and keeps internal/otelforward.Registry's per-EG forwarders in sync with each EG's Spec.OTLP. The cm OTLP receiver consults the registry per inbound request to decide whether to ship a copy of the captured signal to the customer's endpoint.

Cm always persists captured signals to the embedded ClickHouse via internal/chwriter regardless of forwarder state, so a missing or wedged forwarder never costs us the durable copy.

`clrk dev` doesn't go through this reconciler — the cm OTLP receiver is wired with an unconditional dev sink (see cmd/controller-manager/main.go, --dev-otlp-fallback-endpoint) that fans every inbound payload to the TUI receiver regardless of EG state. That way the TUI lights up even when no EgressGateway is applied.

func (*EgressGatewayOTLPForwardReconciler) Reconcile

Reconcile applies the EG's Spec.OTLP to the registry: install or swap when the endpoint changes, remove when the EG is deleted or the endpoint is cleared. Spec-equality short-circuiting lives inside Registry.Apply.

func (*EgressGatewayOTLPForwardReconciler) SetupWithManager

func (r *EgressGatewayOTLPForwardReconciler) SetupWithManager(mgr ctrl.Manager) error

SetupWithManager wires the reconciler onto EgressGateway events. No predicate: change detection happens inside Registry.Apply via the spec-DeepEqual short-circuit, so EG status churn is filtered without predicate boilerplate.

type EgressGatewayReconciler

type EgressGatewayReconciler struct {
	client.Client
	Scheme *runtime.Scheme

	// EnvoyImage is the container image used for the EG-managed Envoy
	// fleet. Must contain the grpc_certificate_provider handshaker
	// extension. Set from --envoy-image on controller-manager.
	EnvoyImage string

	// DevBackendHost, when non-empty, overrides
	// Status.Listeners[*].BackendAddress to use this host with the EG
	// Service NodePort instead of the in-cluster DNS name. Set by
	// `clrk dev` to the docker hostname of the k3s container ("clrk-k3s")
	// because workers run on the docker network and can't route to k3s
	// ClusterIPs.
	DevBackendHost string

	// EnvoyGatewayNamespace is the namespace EG provisions per-EG
	// Envoy Deployments + Services into. Equal to the controller-
	// manager's runtime namespace (mirrors EG's
	// ENVOY_GATEWAY_NAMESPACE env var). Used here to mirror
	// upstream-CA Secrets and to list EG-managed Services for
	// listener status. Required.
	EnvoyGatewayNamespace string

	// InsecureGRPC mirrors controller-manager's --insecure-grpc-no-mtls.
	// When true (dev), the :9443 control plane is plaintext and identity is
	// derived from :authority, so the per-EG client cert + CA mounts are
	// skipped. When false (the secure default), the reconciler mints a
	// per-EG client cert and the Envoy Deployment mounts it plus the shared
	// control-plane CA.
	InsecureGRPC bool

	// ControlPlaneRootCert / ControlPlaneRootKey are the control-plane root
	// CA the controller-manager minted at startup (see controlplane_pki.go).
	// Held in memory so per-EG client certs mint without a cross-namespace
	// read of the root private key. Empty when InsecureGRPC is true.
	ControlPlaneRootCert []byte
	ControlPlaneRootKey  []byte
}

EgressGatewayReconciler provisions Envoy Gateway infrastructure for each clrk EgressGateway: a shared GatewayClass, an EnvoyProxy referencing our private Envoy image, a Gateway with one TCP listener, and the per-EG CA Secret consumed by both the sandbox trust store and the handshaker.

func (*EgressGatewayReconciler) Reconcile

func (r *EgressGatewayReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

func (*EgressGatewayReconciler) SetupWithManager

func (r *EgressGatewayReconciler) SetupWithManager(mgr ctrl.Manager) error

type FallbackRoutingPolicyStatusReconciler

type FallbackRoutingPolicyStatusReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

FallbackRoutingPolicyStatusReconciler writes the GEP-2649 PolicyStatus of a FallbackRoutingPolicy: one ancestor per targetRef with an Accepted condition reporting whether the (same-namespace) AIProviderRoute target exists. It is status-only — the egextension reads the policy off its spec when synthesizing the per-rule clusters — so a dangling targetRef surfaces as Accepted=False/TargetNotFound instead of silently never applying fallback.

func (*FallbackRoutingPolicyStatusReconciler) Reconcile

func (*FallbackRoutingPolicyStatusReconciler) SetupWithManager

func (r *FallbackRoutingPolicyStatusReconciler) SetupWithManager(mgr ctrl.Manager) error

type HTTPInvoker

type HTTPInvoker struct {
	Client client.Client
	HTTP   *http.Client
}

HTTPInvoker invokes a TaskAgent by POSTing directly to the WorkerPool's in-cluster dispatch Service. Reads the Service's EndpointSlices and dials a ready pod IP rather than the Service DNS — the controller-manager runs outside the cluster in `clrk dev` and so can't resolve `*.svc.cluster.local`, but the docker bridge / pod IPs in EndpointSlices are routable in both modes (production: shared pod CIDR; dev: shared docker network via the selectorless Service+EndpointSlice bridge that `clrk dev` materializes for the default WorkerPool).

func NewHTTPInvoker

func NewHTTPInvoker(c client.Client) *HTTPInvoker

NewHTTPInvoker returns an HTTPInvoker with a default http.Client. The per-call deadline is supplied by the caller's context, so the client itself has no Timeout set.

func (*HTTPInvoker) Invoke

func (h *HTTPInvoker) Invoke(ctx context.Context, ta *clrkv1alpha1.TaskAgent, body []byte) error

type InvocationActiveCounter

type InvocationActiveCounter interface {
	CountActive(ctx context.Context, namespace string, kind clrkv1alpha1.InvocationParentKind, name string) (int32, error)
}

InvocationActiveCounter reports the number of non-terminal invocations for a parent agent from the invocation read model. Injected by the controller-manager (backed by invocation.ActiveCounter). Defined here so the controller package needn't import the apiserver/ClickHouse stack; nil in tests / no-ClickHouse builds, where the reconciler falls back to the per-worker WorkerStatus sum.

type TaskAgentCronReconciler

type TaskAgentCronReconciler struct {
	client.Client
	Scheme  *runtime.Scheme
	Invoker TaskAgentInvoker
	// Now is overridable so tests can advance the clock without sleeping.
	Now func() time.Time
	// contains filtered or unexported fields
}

TaskAgentCronReconciler watches TaskAgents with a non-empty spec.schedule and fires them on a cron clock via Invoker. The fire loop runs as a separate Runnable that opts into leader election.

func (*TaskAgentCronReconciler) Reconcile

func (r *TaskAgentCronReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

func (*TaskAgentCronReconciler) SetupWithManager

func (r *TaskAgentCronReconciler) SetupWithManager(mgr ctrl.Manager) error

SetupWithManager registers the reconciler and the tick runnable on mgr.

func (*TaskAgentCronReconciler) Tick

Tick walks all registered entries and fires those whose nextFire time has passed. Fires within a single tick run in parallel goroutines so one slow Invoker can't starve sibling agents; Tick returns once all in-flight fires complete (so the next ticker tick is paced to the slowest fire — good enough for minute-resolution cron).

Exported because tests drive it directly to avoid sleeping.

type TaskAgentIngressReconciler

type TaskAgentIngressReconciler struct {
	client.Client
	Scheme *runtime.Scheme

	// IngressExtProcHost / IngressExtProcPort point at the
	// controller-manager's ingress ext_proc gRPC endpoint. Wired from
	// `--ingress-extproc-host` / `--ingress-extproc-port`.
	IngressExtProcHost string
	IngressExtProcPort int32
}

TaskAgentIngressReconciler owns the k8s-side half of TaskAgent.

Per-TaskAgent it materializes:

  • a Gateway (one HTTP listener) and HTTPRoute (per-TA path tree)
  • an EG `Backend` of `type: DynamicResolver` — EG generates a dynamic_forward_proxy cluster that dials whatever `:authority` the request carries, so the HTTPRoute backendRef is this object instead of a static Service
  • an `EnvoyExtensionPolicy` attaching the controller-manager's ingress ext_proc onto the per-TA HTTPRoute

Per TaskAgent's namespace it also materializes a shared `clrk-ingress-extproc` Backend pointing at the controller-manager's ingress ext_proc gRPC service (FQDN + port from IngressExtProc{Host,Port}). One Backend per namespace keeps the ExtensionPolicy backendRef short and re-targetable from a single place.

func (*TaskAgentIngressReconciler) Reconcile

func (*TaskAgentIngressReconciler) SetupWithManager

func (r *TaskAgentIngressReconciler) SetupWithManager(mgr ctrl.Manager) error

type TaskAgentInvoker

type TaskAgentInvoker interface {
	Invoke(ctx context.Context, ta *clrkv1alpha1.TaskAgent, body []byte) error
}

TaskAgentInvoker fires one execution of ta. Implementations route to the same path an HTTP-triggered TaskAgent would take. body is the verbatim scheduleInput (default empty JSON object when nil).

type TaskAgentRevisionReconciler

type TaskAgentRevisionReconciler struct {
	client.Client
	Scheme *runtime.Scheme

	// ActiveExec counts non-terminal invocations for the source of
	// TaskAgent.Status.ActiveExecutions (APO-620). Nil falls back to the
	// legacy per-worker sum off AgentSandboxRevision.Status.Workers — kept
	// for the embedded-ClickHouse binary-layering rollout window, after
	// which the fallback can be dropped.
	ActiveExec InvocationActiveCounter

	// Recorder emits rollout notifications (events.k8s.io/v1) on the
	// RevisionReady False->True crossing. Nil-safe (tests / no-apiserver builds).
	Recorder *notify.Recorder
}

TaskAgentRevisionReconciler manages AgentSandboxRevision snapshots for a TaskAgent. Gateway / HTTPRoute live in TaskAgentIngressReconciler so they can be skipped where gateway-api isn't installed.

func (*TaskAgentRevisionReconciler) Reconcile

func (*TaskAgentRevisionReconciler) SetupWithManager

func (r *TaskAgentRevisionReconciler) SetupWithManager(mgr ctrl.Manager) error

type WorkerPoolDeploymentReconciler

type WorkerPoolDeploymentReconciler struct {
	client.Client
	Scheme *runtime.Scheme

	// CMOTLPEndpoint is the OTLP/HTTP URL of the controller-manager's
	// receiver. The reconciler stamps it on every worker container as
	// CLRK_CM_OTLP_ENDPOINT so the worker's OTLP emitter knows where
	// to ship signals. Empty disables injection — the worker emitter
	// falls back to noop.
	CMOTLPEndpoint string

	// CMNATSAddr is the controller-manager's NATS/JetStream client
	// address (host:port). The reconciler stamps it on every worker
	// container as CLRK_CM_NATS_ADDR so the dispatcher's invocation
	// publisher can dial the cm's embedded JetStream over TCP. Empty
	// disables injection — the worker simply doesn't publish lifecycle
	// events.
	CMNATSAddr string

	// Recorder emits fleet-health notifications (events.k8s.io/v1) when a
	// WorkerPool crosses the ready>=desired health boundary. Nil-safe.
	Recorder *notify.Recorder
}

WorkerPoolDeploymentReconciler owns the k8s-side half of WorkerPool: it creates/updates the Deployment + Service that host worker pods and reports their health back as ReadyReplicas + Available/Progressing conditions.

func (*WorkerPoolDeploymentReconciler) Reconcile

func (*WorkerPoolDeploymentReconciler) SetupWithManager

func (r *WorkerPoolDeploymentReconciler) SetupWithManager(mgr ctrl.Manager) error

type WorkerPoolStatusReconciler

type WorkerPoolStatusReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

WorkerPoolStatusReconciler owns the portable half of WorkerPool: it aggregates execution capacity and per-agent active-execution counts, and writes the "Configured" condition. It never reads or writes Deployment / Service — that lives in WorkerPoolDeploymentReconciler. The `ReadyReplicas` and `Progressing` fields are left to that reconciler.

func (*WorkerPoolStatusReconciler) Reconcile

func (*WorkerPoolStatusReconciler) SetupWithManager

func (r *WorkerPoolStatusReconciler) SetupWithManager(mgr ctrl.Manager) error

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址