Central orchestrator API for the Lattice container-orchestration platform — workers, stacks, containers, deployments, registries, database instances, and backups across a fleet of Docker hosts.
appleby.cloud platform · Go API ·
lattice.appleby.cloud
lattice-api is the brain of Lattice (an in-house replacement for Portainer). It is a monolithic Go HTTP + WebSocket server that holds all persistent state in a MariaDB lattice database and is the only component that can issue commands to workers. Each host runs a lattice-runner agent that dials outbound to /ws/worker; the API pushes deploy/start/stop/rollback and database commands down that socket and ingests heartbeats, metrics, and lifecycle telemetry back. The Next.js dashboard talks to it over REST plus a second admin WebSocket (/ws/admin) for live updates.
It owns the data model, command dispatch, deployment bookkeeping (record + watchdog that pings, retries, and force-fails stalled deploys), auth/RBAC, fleet observability plumbing (health scanning, metrics, anomalies, retention, webhooks, SMTP alerts), and automations — named rules pairing a webhook or cron trigger with an ordered list of actions, so one CI call can redeploy containers across any number of stacks (a deploy token is bound to one), with every firing recorded as a run. It does not perform the Docker work itself — pulling images and executing the blue-green/canary/rolling strategy step-by-step lives in lattice-runner; the API only sends a strategy string and a resolved container spec.
| Repo | Relationship |
|---|---|
lattice-web |
Next.js dashboard — consumes /admin/* REST + /ws/admin. |
lattice-runner |
Agent on each worker VM — dials /ws/worker outbound and executes Docker ops. The worker protocol is a shared contract; change both together. |
lattice-mcp |
MCP server exposing /admin as Claude tools — keep it in sync when routes change. |
Lattice is self-hosting: Forta (SSO), Keyring, Monitor and the rest all run as stacks on Lattice workers, which is why local auth exists as a fallback when the SSO IDP is itself down.
- Go 1.26 with
gorilla/mux(subrouters for/authand/admin) - MariaDB — primary data store (schema
lattice, migrated in-code via idempotentmigrate()calls indb.Init(); no external migration runner) gorilla/websocket— worker hub and admin hubsquirrel(sq) — SQL query builder, no ORMgolang-jwt/jwt/v5— local JWT auth (HS512)- Provider-agnostic SSO — via
go-oidc-session: PKCE S256, single-use server-side state, and a server-side SSO session whose lifetime is the identity provider's (tokens refreshed when due; the session ends when the provider refuses or back-channels a logout). lattice-web re-authenticates silently withprompt=none. Lattice keeps its own users; local password accounts keep their JWTs. rs/corsfor CORS,golang.org/x/crypto(bcrypt) for passwords, AES-256-GCM for secrets at rest
- Go 1.26+
- Docker — for
dev up(bundled MariaDB) and, in production, the Docker-socket self-update path - A MariaDB reachable via
DATABASE_DSN(the base DSN only, no database name —db.Init()appends/lattice) - A
.envfile (copy.env.example); thedevserver sources it
cp .env.example .env # then set DATABASE_DSN and a strong JWT_SIGNING_KEY
dev up # start MariaDB + API + web via docker compose
# or run the API alone against your own MariaDB:
dev # sources .env, go run .Set at least DATABASE_DSN and JWT_SIGNING_KEY (min 32 chars; production panics on weak/known-default keys). In production ENCRYPTION_KEY (64 hex chars) is required — the app panics at boot without it rather than storing secrets as plaintext; in development it may be omitted (loud warning, plaintext passthrough). Set TRUSTED_PROXIES (comma-separated IPs/CIDRs) if the API sits behind a reverse proxy so rate limiting reads the forwarded client IP instead of the proxy's. SSO is optional — uncomment the SSO_* block in .env.example or configure it at runtime via PUT /admin/sso-config. With SSO on, ACTIVITY_REPORT_URL (optional) mirrors SSO users' audited actions to an activity sink — in prod Forta's https://auth.appleby.cloud/internal/activity; see Activity sink in AGENTS.md. Monitor telemetry is optional too, and never a boot requirement: set MONITOR_INGEST_URL and an ingest-scoped MONITOR_API_KEY (plus MONITOR_SPOOL_DIR on a persistent volume) to ship every info-and-above log line (debug too with MONITOR_DEBUG), request, panic and boot failure to the appleby zone — see Monitor telemetry in AGENTS.md. See .env.example for the rest.
| Command | What it does |
|---|---|
dev |
Run the HTTP dev server (source .env && go run .) |
dev dev-https |
Run over HTTPS with local mkcert certs |
dev build |
Build the binary (go build -o bin/app .) |
dev test |
Run tests (go test ./...) |
dev fmt |
Format (gofmt -w -s .) |
dev vet |
Vet (go vet ./...) |
dev check |
fmt + vet + test |
dev tidy |
go mod tidy |
dev up / dev down |
docker compose up -d / down (MariaDB + API + web) |
dev setup-local |
One-time mkcert + /etc/hosts for lattice-api.local.appleby.cloud |
Provisioning a worker — from any host:
curl -fsSL https://lattice-api.appleby.cloud/install/runner | \
REGISTRY_USERNAME=x REGISTRY_PASSWORD=x WORKER_TOKEN=<token> WORKER_NAME=<name> bashThe script (served from GET /install/runner) clones, builds, configures, and installs the runner as a systemd service.
main.go # Entry point — router, all route registrations, middleware, WS endpoints
init.go / server.go # Dependency wiring (hubs, handlers, background jobs) / HTTP server + graceful shutdown
message_handlers.go # Worker & admin WebSocket OnConnect/OnDisconnect/OnMessage callbacks
ws_dispatch.go # Persistence helpers for inbound worker messages
container_cache.go # 60s name→container cache (kills the per-message N+1 lookup)
env/ db/ logger/ # Env vars; MariaDB pool + Queryable + in-code migrations; structured logging
telemetry/ # Monitor wiring — slog handler (Monitor fan-out, warn limiter), panic tee, boot-failure reporting (never a boot requirement)
middleware/ # DualAuth, RejectPending, RequireAdmin/Editor, WorkerTokenAuth, CSRF, rate limit
jwt/ crypto/ sso/ # Local JWTs (HS512); AES-256-GCM secrets; SSO config + go-oidc-session Manager
routers/ # ~80 Handle<Verb><Entity>.router.go handlers (+ deployment monitor, audit helper)
query/ # 27 squirrel query files, all taking db.Queryable first
structs/ # 20 domain types with json tags + pointer nullables
socket/ # WorkerHub / AdminHub, connection handler, protocol constants
registry/ # Docker Registry v2 client (repos, tags, credential test, manifest digest)
healthscan/ watcher/ retention/ webhooks/ versions/ mailer/ # Background subsystems
automations/ # Automation executor — webhook/cron trigger → ordered steps, run records, run-time authorisation
cron/ # The one 5-field cron evaluator + validator (snapshot scheduler, staleness alarm, automations)
bootstrap/ tools/ # First-run admin creation; password/token hashing + validators
See AGENTS.md for the full route table, auth model, worker protocol, and deploy mechanics.
Deployed on Lattice itself at lattice.appleby.cloud, as the lattice-api container in its stack (alongside lattice-web and a docker-helper). It mounts the Docker socket because POST /admin/update/api|web shells out to docker compose in DOCKER_COMPOSE_DIR to pull :latest and recreate the API/web containers.
CI (.github/workflows/build-and-deploy.yml): on push to main it resolves a version (a commit-message [release-patch|minor|major] tag bumps the latest vX.Y.Z git tag, pushes :latest + :vX.Y.Z + a GitHub release; otherwise :development + :<short-sha>), injects registry credentials via keyring-actions, then builds and pushes to registry.appleby.cloud/lattice-api. CI builds and pushes the image; rollout happens through Lattice's own self-update path. The API version is stamped at build time via -ldflags "-X main.Version=<tag>" and exposed at GET /version.
AGENTS.md— the authoritative deep reference (full route surface, dual-auth model, WebSocket hubs, worker protocol, deploy strategies, schema, operations, guardrails).- Related repos:
lattice-web,lattice-runner,lattice-mcp. - Verify before "done":
gofmt -l .(clean),go build ./...,go vet ./...,go test ./...(ordev check).