The event ingestion, query, and observability API for the Monitor platform — a high-performance service that ingests monitoring events into ClickHouse and owns everything relational in MariaDB: identity/authentication (native accounts + pluggable SSO), the tenancy registry, issue tracking, and the alerting and dashboard configuration.
Monitor platform · Go API ·
monitor.appleby.cloud(Lattice)
go services (go-monitor / monitor-js SDK) monitor-web (Next.js)
↓ (batched NDJSON over HTTP, X-Api-Key) ↓ (proxy: mon-* cookies + X-CSRF-Token)
monitor-core ──────────────────────────────────────────────────────────
↓ (batched inserts) ↓ (rows: identity, tenancy, issues, alert + dashboard config)
ClickHouse (events) MariaDB (monitor_auth + monitor)
Which store owns what — the rule is facts vs configuration. ClickHouse holds rows
written by machines, appended at volume and expired by a TTL: events,
issue_occurrences_daily, alert_states, alert_history. MariaDB holds rows a human
edits and whose correctness depends on constraints: users, api keys, zones/projects,
issues, service→repo mappings, and — since migrations 119-124 — alert rules, notification
channels, notification policies, service groups, dashboards and saved views, all seven
project-scoped since 127-133. See
AGENTS.md §6 Stores — configuration vs facts for the argument and for the
two production defects that produced it.
- HTTP ingestion endpoint:
POST /v1/eventsaccepts NDJSON (newline-delimited JSON) - Gzip support: Automatically handles gzip-compressed request bodies, and gzips
/v1JSON responses of 1 KB or more when the client sendsAccept-Encoding: gzip - Streaming parser: Processes events line-by-line without loading entire body into memory
- Batched writes: Collects events and writes to ClickHouse in configurable batches
- Non-blocking ingestion: HTTP handler enqueues events and returns immediately
- Query + analytics API: search, autocomplete, aggregation, time series, top-N, gauge, compare
- Alerting, issues, dashboards, saved views and two SSE streams (live tail + alert feed)
- Native authentication: email+password accounts (bcrypt 12) + Monitor-owned HS512 JWT sessions (15m access / rotating 7d refresh with reuse detection)
- Pluggable SSO: config-driven OIDC/OAuth2 providers with true account linking (one user, many identities) — any IdP is a config row, not a code change
- API-key authentication for ingest/query: via the
X-Api-Keyheader
dev updev up brings up both datastores from docker-compose.yml: ClickHouse (events) and
MariaDB (monitor_auth, host port 3336 — the auth data layer).
Both schemas are applied automatically at startup, fail-fast:
- ClickHouse —
migrations/embed.goembedsmigrations/*.sql(idempotentIF NOT EXISTS), run frommain.goright after connecting. The DDL is written against the literal databasemonitorand rewritten toCLICKHOUSE_DATABASEbefore it runs, so the migrated database is always the one being served;main.gothen reads theeventstable once and refuses to start if it can't. - MariaDB —
db/sql.go'sdb.RunMigrations()embedsdb/migrations/*.sql, runs each once, tracked in amigrations_appliedtable.
A fresh deploy self-migrates — no manual step is required. It then seeds the first admin
user and the tenancy registry (one zone, one project — MON_ZONE_SLUG /
MON_DEFAULT_PROJECT), both no-ops once the rows exist.
migrations/manual/ is the exception: one-off reconciliation SQL that is not embedded,
not rewritten for CLICKHOUSE_DATABASE, and run by hand. Each file's header carries its
own before/watch/after queries; the operator runbook for all of them — what each does, when
to run it, and how to confirm the mutation landed — is AGENTS.md §8.
MONITOR_API_KEY is required — the master ingest/query key (header X-Api-Key); the
server refuses to start without it. In a production profile (MON_COOKIE_INSECURE
unset/false) the server also refuses to start unless MON_JWT_SIGNING_KEY and a 32-byte
MON_CRYPTO_KEY are set to non-default values.
# Ingest/query
export MONITOR_API_KEY="your-secret-key"
# Auth (local dev — allows the dev-default session/crypto keys)
export MON_COOKIE_INSECURE=true
export MON_DB_DSN="monitor:monitor@tcp(127.0.0.1:3336)/monitor_auth"
export MON_ADMIN_EMAIL="you@example.com" # seeds the first admin on a fresh DB
export MON_ADMIN_PASSWORD="a-strong-password"To sign in with SSO locally, add a provider via POST /admin/sso-providers (or the
/admin/sso page in monitor-web). See the full env table under
Configuration and the auth model in
AGENTS.md §6. All other defaults work with dev up.
dev runOr build and run:
go build -o bin/monitor-core .
./bin/monitor-coredocker-compose up -dcurl http://localhost:8080/healthResponse:
status is ok, or degraded with a failing array naming what this role needs and is
not getting. The HTTP code stays 200 either way — the container HEALTHCHECK polls this path,
and restarting the process cannot repair a datastore. GET /ready is the endpoint that
answers 503.
{
"status": "ok",
"enqueued": 0,
"dropped": 0,
"pending": 0,
"clickhouse_ok": true,
"clickhouse_writer_ok": true,
"mariadb_ok": true,
"last_flush_at": "2026-09-06T12:00:00Z",
"role": "both",
"zone": "trailblaze",
"alerting_ok": true,
"telemetry": {
"shipping": true,
"enqueued": 412,
"flushed": 410,
"dropped": 0,
"quarantined": 0,
"pending": 0,
"last_drop_at": null
}
}clickhouse_writer_ok is the batcher's own ClickHouse pool (2 connections, separate from the
10-connection read pool so dashboard reads cannot starve ingest flushes). false beside a
true clickhouse_ok means reads answer but ingest cannot write; failing then names
clickhouse_writer. If that pool could not be opened at boot, writes share the read pool and
this flag mirrors clickhouse_ok.
telemetry is monitor-core's own telemetry shipper — the events it reports about itself
to the appleby zone (see Self-telemetry). flushed rising is healthy;
pending rising with flushed still means the destination is down; dropped is loss.
/health is liveness — it always returns 200 / "ok", even with a store down (the
container healthcheck points here, and a restart cannot repair ClickHouse). The
dependency booleans and last_flush_at are diagnostics; dropped counts both queue
overflow and batches the writer gave up on. role echoes MON_ROLE (see Roles under
Configuration) so the plane a process thinks it is can be asked rather than inferred.
zone echoes MON_ZONE_SLUG — this process's identity. It is what makes
POST /admin/zones/{id}/probe able to tell "a monitor-core answered" from "the
monitor-core I meant answered": every zone runs this same binary and every one of them
answers 200, so without it a registry row pointed at the wrong box looks perfectly
healthy while every read through it returns another tenant's data. Read role alongside
it — a control plane carries a zone slug too and serves no events at all.
curl -i http://localhost:8080/ready200 when ClickHouse (both its read pool and the batcher's writer pool) and MariaDB all
answer a 2s ping, otherwise 503:
{ "status": "not_ready", "clickhouse_ok": false, "clickhouse_writer_ok": false, "mariadb_ok": true, "role": "both", "failing": ["clickhouse", "clickhouse_writer"] }Under MON_ROLE=app there is no ClickHouse connection at all, so it is not counted as a
dependency and clickhouse_ok: false does not make the process un-ready — that is what
role on this response is for. Every other role still requires it.
curl -X POST http://localhost:8080/v1/events \
-H "Content-Type: application/x-ndjson" \
-H "X-Api-Key: your-secret-key" \
-d '{"timestamp":"2026-02-06T23:01:02.123Z","service":"users","job_id":"job_x","request_id":"req_y","trace_id":"trc_z","name":"user.created","data":{"user_id":42}}
{"timestamp":"2026-02-06T23:01:02.456Z","service":"users","job_id":"job_x","request_id":"req_y","trace_id":"trc_z","name":"db.query","data":{"table":"users"}}'Response:
{ "accepted": 2 }Each event must be a JSON object on its own line with these fields:
| Field | Type | Required | Description |
|---|---|---|---|
timestamp |
string (RFC3339) | Yes | When the event occurred |
service |
string | Yes | Service name that generated the event |
name |
string | Yes | Event type/name |
env |
string | No | Environment (e.g., production, staging) |
job_id |
string | No | Groups related requests within a service |
request_id |
string | No | Unique identifier per incoming request |
trace_id |
string | No | Spans across services for distributed tracing |
user_id |
string | No | User identifier for user-scoped queries |
level |
string | No | Log level (info, warn, error, debug) |
data |
object | No | Additional event data |
Query events with filters (Grafana-style):
curl "http://localhost:8080/v1/events?service=users&level=error&limit=50" \
-H "X-Api-Key: your-secret-key"Query Parameters:
| Parameter | Description |
|---|---|
service |
Filter by service name |
env |
Filter by environment |
job_id |
Filter by job ID |
request_id |
Filter by request ID |
trace_id |
Filter by trace ID |
user_id |
Filter by user ID |
name |
Filter by event name |
level |
Filter by log level |
from |
Start time (RFC3339 or Unix timestamp) |
to |
End time (RFC3339 or Unix timestamp) |
data.<key> |
Filter by data field (e.g., data.user_id=42) |
limit |
Results per page (default: 100, max: 1000) |
offset |
Pagination offset |
Filter Operators:
Filters support operators using Django-style syntax: field__operator=value
| Operator | Example | Description |
|---|---|---|
eq |
service=users or service__eq |
Equals (default) |
neq |
level__neq=debug |
Not equals |
lt |
data.count__lt=100 |
Less than |
gt |
data.count__gt=10 |
Greater than |
lte |
data.latency__lte=500 |
Less than or equal |
gte |
data.latency__gte=100 |
Greater than or equal |
contains |
name__contains=user |
Contains substring |
startswith |
service__startswith=auth |
Starts with |
endswith |
name__endswith=.error |
Ends with |
in |
level__in=error,warn |
Matches any (comma-sep) |
Examples:
# Find errors and warnings
curl "http://localhost:8080/v1/events?level__in=error,warn"
# Find events with latency > 500ms
curl "http://localhost:8080/v1/events?data.latency_ms__gt=500"
# Find user-related events
curl "http://localhost:8080/v1/events?name__contains=user"
# Exclude debug logs
curl "http://localhost:8080/v1/events?level__neq=debug"Response:
{
"success": true,
"message": "request was successful",
"pagination": { "count": 150, "next": "/v1/events?offset=100&limit=100", "previous": "" },
"data": [{ "timestamp": "...", "service": "users", ... }]
}Get distinct values for a label (service, env, name, level):
curl "http://localhost:8080/v1/labels/service/values" \
-H "X-Api-Key: your-secret-key"Response:
{
"success": true,
"message": "request was successful",
"data": ["users", "orders", "payments"]
}Get available keys from the data JSON column:
curl "http://localhost:8080/v1/data/keys?service=users" \
-H "X-Api-Key: your-secret-key"Response:
{
"success": true,
"message": "request was successful",
"data": ["client_ip", "host", "method", "path", "user_agent"]
}Get values for a specific data key:
curl "http://localhost:8080/v1/data/values?key=method&service=users" \
-H "X-Api-Key: your-secret-key"Response:
{
"success": true,
"message": "request was successful",
"data": ["GET", "POST", "PUT", "DELETE"]
}The analytics API provides Grafana-compatible endpoints for building dashboards, charts, and gauges.
Aggregate data with optional grouping:
# Count events grouped by service
curl -X POST "http://localhost:8080/v1/analytics" \
-H "Content-Type: application/json" \
-H "X-Api-Key: your-secret-key" \
-d '{
"aggregation": "count",
"group_by": ["service"],
"from": "2026-02-01T00:00:00Z",
"to": "2026-02-06T23:59:59Z"
}'Request Body:
| Field | Type | Required | Description |
|---|---|---|---|
aggregation |
string | No | Aggregation type (default: count) |
field |
string | * | Field to aggregate (required for sum/avg/min/max/percentiles) |
group_by |
string[] | No | Fields to group by (max 10) |
filters |
object[] | No | Filter conditions |
from |
string | No | Start time (RFC3339 or Unix) |
to |
string | No | End time (RFC3339 or Unix) |
order_by |
string | No | Field to order by (value or group field) |
order_desc |
boolean | No | Order descending |
limit |
integer | No | Max results (default: 100, max: 10000) |
Aggregation Types:
| Type | Description | Requires Field |
|---|---|---|
count |
Count of events | No |
count_unique |
Count of unique values | Yes |
sum |
Sum of numeric field | Yes |
avg |
Average of numeric field | Yes |
min |
Minimum value | Yes |
max |
Maximum value | Yes |
p50 |
50th percentile | Yes |
p90 |
90th percentile | Yes |
p95 |
95th percentile | Yes |
p99 |
99th percentile | Yes |
Filter Format:
{
"filters": [
{ "field": "service", "operator": "eq", "value": "users" },
{ "field": "data.status_code", "operator": "gte", "value": 400 }
]
}Response:
{
"success": true,
"data": {
"data": [
{ "value": 1523, "groups": { "service": "users" } },
{ "value": 892, "groups": { "service": "orders" } }
],
"total": 2
}
}GET endpoint (query-string based):
curl "http://localhost:8080/v1/analytics?aggregation=count&group_by=service&from=2026-02-01T00:00:00Z"Get time-bucketed data for charts:
curl -X POST "http://localhost:8080/v1/timeseries" \
-H "Content-Type: application/json" \
-H "X-Api-Key: your-secret-key" \
-d '{
"aggregation": "count",
"interval": "hour",
"filters": [{ "field": "name", "operator": "eq", "value": "user.login" }],
"from": "2026-02-05T00:00:00Z",
"to": "2026-02-06T23:59:59Z",
"fill_zeros": true
}'Request Body:
| Field | Type | Required | Description |
|---|---|---|---|
aggregation |
string | No | Aggregation type (default: count) |
field |
string | * | Field to aggregate |
interval |
string | Yes | Time bucket size |
group_by |
string[] | No | Fields to group by (creates multiple series) |
filters |
object[] | No | Filter conditions |
from |
string | No | Start time |
to |
string | No | End time |
fill_zeros |
boolean | No | Fill empty buckets with zero |
Interval Types: minute, hour, day, week, month
Response:
{
"success": true,
"data": {
"series": [
{
"name": "",
"data_points": [
{ "timestamp": "2026-02-05T00:00:00Z", "value": 42 },
{ "timestamp": "2026-02-05T01:00:00Z", "value": 38 }
]
}
]
}
}GET endpoint:
curl "http://localhost:8080/v1/timeseries?interval=hour&name=user.login&fill_zeros=true"Get top N values for a dimension:
curl -X POST "http://localhost:8080/v1/topn" \
-H "Content-Type: application/json" \
-H "X-Api-Key: your-secret-key" \
-d '{
"aggregation": "count",
"group_by": "data.endpoint",
"limit": 10,
"from": "2026-02-01T00:00:00Z",
"to": "2026-02-06T23:59:59Z"
}'Response:
{
"success": true,
"data": {
"data": [
{ "key": "/api/users", "value": 5234 },
{ "key": "/api/orders", "value": 3891 }
]
}
}Get a single aggregated value:
curl -X POST "http://localhost:8080/v1/gauge" \
-H "Content-Type: application/json" \
-H "X-Api-Key: your-secret-key" \
-d '{
"aggregation": "count",
"filters": [{ "field": "level", "operator": "eq", "value": "error" }],
"from": "2026-02-06T00:00:00Z",
"to": "2026-02-06T23:59:59Z"
}'Response:
{
"success": true,
"data": { "value": 127 }
}Compare current period with a previous period:
curl -X POST "http://localhost:8080/v1/compare" \
-H "Content-Type: application/json" \
-H "X-Api-Key: your-secret-key" \
-d '{
"aggregation": "count",
"filters": [{ "field": "name", "operator": "eq", "value": "http.request" }],
"from": "2026-02-06T00:00:00Z",
"to": "2026-02-06T23:59:59Z"
}'Response:
{
"success": true,
"data": {
"current": 1523,
"previous": 1342,
"change": 181,
"change_percent": 13.49
}
}If compare_from/compare_to are not specified, the previous period is auto-calculated based on the duration of the current period.
Monitor owns identity end-to-end — it delegates to no external identity provider. Two credential kinds coexist:
- API keys (
X-Api-Key) — for ingestion (POST /v1/events,ingestscope only — anadminkey is a query credential and is refused here with a401) and machine query callers (adminscope). The env master key isMONITOR_API_KEYand still does both. Revoking a key takes effect within 30s, the cache refresh interval. Every key also binds to a project (api_keys.project_id,NOT NULL), which is where ingestion derives an event's tenant from;POST /v1/api-keysaccepts an optionalproject_slugand defaults toMON_DEFAULT_PROJECT. Key names are unique per project, so each project may own aningestkey — but the key hash stays unique instance-wide, because a presented key must resolve to exactly one row. That binding also decides what anadminkey can read: every event query is scoped to the key's own project (see below). The env master key and dashboard sessions readMON_DEFAULT_PROJECT. - Sessions — for the
monitor-webdashboard. Native email+password accounts (bcrypt cost 12) or pluggable SSO mint a Monitor-owned HS512 JWT session: a 15-minute access token and a rotating 7-day refresh token with reuse detection. Delivered as themon-*cookies (mon-access-token,mon-refresh-token,mon-logged-in,mon-csrf); unsafe requests are CSRF-protected via a double-submitmon-csrf↔X-CSRF-Tokencheck. Roles gate/v1for sessions: apendingaccount is refused everything (403,error_code 4004), aviewermay read but not write (403,4003),editorandadminmay do both. Comment edits/deletes are limited to the author or an admin. Key use is recorded inapi_keys.last_used_at(flushed once a minute).
Auth/identity data lives in MariaDB (monitor_auth): users, identities
(UNIQUE(provider, provider_user_id) — true account linking, one user ↔ many sign-in
methods), refresh_tokens, sso_providers, oidc_sessions, settings, api_keys, plus
the tenancy registry (zones, projects). The second MariaDB schema, monitor, holds
observability data on the same connection: issues, issue_timeline, issue_links,
service_repos, and the alerting/dashboard configuration (alert_rules,
notification_channels, notification_policies, service_groups, dashboards,
saved_views).
Endpoint surface (browser/cookie-oriented, outside /v1):
POST /auth/login | /auth/register | /auth/refresh | /auth/logout
GET/PUT /auth/self
GET /auth/self/identities POST/DELETE /auth/self/identities/{slug}
GET /auth/sso/config | /auth/sso/{slug}/login (?prompt=none&return_url=) | /auth/sso/callback
POST /auth/sso/{slug}/backchannel-logout (OIDC Back-Channel Logout; public)
GET/POST /admin/sso-providers PUT/DELETE /admin/sso-providers/{slug} (admin only)
The SSO protocol itself lives in go-oidc-session
— this repo keeps only the parts that library refuses to know: the provider-row mapping, secret
resolution, state storage, and the identity-resolution rules. An SSO sign-in opens a server-side
SSO session (mon_sso_session, provider tokens encrypted in oidc_sessions); every
/auth/refresh refreshes it against the provider, so a Monitor session lasts exactly as long as
the provider session and ends within one access-token lifetime of a revocation.
SSO providers are config rows (OIDC via discovery, or explicit-URL OAuth2). Adding one —
Google, Okta, Entra, Forta, anything else — is a POST /admin/sso-providers, never a
build. Linking is nOAuth-safe: identity is keyed on (provider, subject), and
link-on-login only fires when both the IdP email and the existing account email are
verified. Full details in AGENTS.md §6.
| Environment Variable | Default | Description |
|---|---|---|
HTTP_PORT |
8080 |
HTTP server port |
MON_ROLE |
both |
Which plane this process runs: app (control plane — identity/config, no ClickHouse), zone (data plane — events, ingest, alerts, issues) or both. Leave it unset unless you are deliberately splitting the planes; an unrecognised value refuses to boot rather than falling back. See Roles below |
CLICKHOUSE_ADDR |
localhost:9000 |
ClickHouse server address |
CLICKHOUSE_DATABASE |
monitor |
ClickHouse database name; must match ^[a-z][a-z0-9_]{2,62}$ (it is interpolated into SQL, and the migrations are rewritten to it) |
CLICKHOUSE_USERNAME |
default |
ClickHouse username |
CLICKHOUSE_PASSWORD |
`` | ClickHouse password |
CLICKHOUSE_MAX_MEMORY_USAGE |
2147483648 |
Per-query memory ceiling in bytes (2 GiB), sent as the max_memory_usage setting. ClickHouse 24.3's own default is unlimited, so unset, the only bound is the server-wide max_server_memory_usage (0.9 × the container's memory limit, or host RAM without one), and one high-cardinality GROUP BY can fail every query — ingest included — or push a shared host into the OOM killer |
MONITOR_API_KEY |
(required) | Master API key (header X-Api-Key); server refuses to start if unset |
BATCH_SIZE |
1000 |
Number of events per batch insert |
ACTIVITY_REPORT_URL / ACTIVITY_REPORT_PROVIDER |
`` | Activity sink (both required): changes by users of SSO provider <slug> are mirrored to the URL. Prod: https://auth.appleby.cloud/internal/activity + forta. Empty: off |
FLUSH_INTERVAL |
5s |
Max time to wait before flushing batch; plus 3 s, also the alert evaluator's settle delay |
QUEUE_SIZE |
100000 |
Max events in memory queue |
MON_DB_DSN |
monitor:monitor@tcp(127.0.0.1:3336)/monitor_auth |
MariaDB DSN (auth data layer); from Keyring in prod |
MON_JWT_SIGNING_KEY |
(dev default) | HS512 session-token key; prod must override |
MON_CRYPTO_KEY |
(dev default) | Exactly 32 bytes — AES-256-GCM key for SSO secrets/tokens; prod must override |
MON_COOKIE_DOMAIN |
`` | Domain set on the mon-* cookies |
MON_COOKIE_INSECURE |
false |
true for local dev: drops Secure, allows dev-default secrets |
MON_PUBLIC_URL |
(none — required) | This process's own public origin. Builds each SSO redirect_uri (must match the IdP registration) and seeds a fresh zone's ingest_url/query_url. Previously defaulted to the control plane's URL, which let a zone register a row pointing at the wrong box |
MON_ADMIN_EMAIL / MON_ADMIN_PASSWORD |
`` | Seed the first admin on a fresh DB; empty = no seed |
MON_ALLOW_REGISTRATION |
false |
Gates POST /auth/register (self-registration) |
MON_ZONE_SLUG |
trailblaze |
The zone this process is. Required on MON_ROLE=zone: a data plane that inherits the fallback seeds a second zone under the control plane's name and reports it on /health. Slugs are immutable — changing it later seeds a new zone, it does not rename the first |
MON_DEFAULT_PROJECT |
default |
Slug of the project seeded inside that zone; every API key binds to it, the env master key stamps events with it, and it is the project the master key and dashboard sessions read. Also the only project whose reads still match pre-006 events (see Tenancy). Not boot-only — do not change it on a running install: it is read on every request, query, streamed event and error fingerprint, and repointing it hides every event the manual backfill has not stamped |
MON_TELEMETRY_INGEST_URL |
(none) | Where monitor-core reports its own errors and activity — the appleby zone's full ingest endpoint. Unset = nothing shipped; warnings and errors still print to stdout |
MON_TELEMETRY_API_KEY |
(none) | Ingest-scope key minted on that zone. MONITOR_API_KEY, which is this service's own master key |
MON_TELEMETRY_ENV / MON_TELEMETRY_ZONE |
production / (none) |
Event env; expected destination zone (appleby), asserted at boot, never sent |
MON_TELEMETRY_SPOOL_DIR |
(none) | Durable telemetry spool (a volume) so events survive the destination being down or the process restarting |
MON_TELEMETRY_DEBUG / MON_TELEMETRY_STDOUT |
false / false |
Debug events; print every event as NDJSON (local verification) |
Both monitor-core processes — the control plane and appleby-core — report their own
errors, warnings and activity to Monitor's appleby zone as service monitor-core. The
MON_TELEMETRY_* variables are read from the plain environment, before Keyring, and none of
them is required: telemetry never blocks or fails the boot. appleby-core reports into
itself, so a successful ingest emits nothing and every self-feeding failure is coalesced to
one event a minute; warnings and errors always print to stdout, which is the only place the
appleby zone's own datastore failures are visible while they last. Destination, loop guards,
level policy and the full event catalogue: AGENTS.md §6 Monitor
self-telemetry.
monitor-core does two unrelated jobs in one binary, and MON_ROLE says which of them a
process is running.
app |
zone |
both (default) |
|
|---|---|---|---|
Identity: /auth/*, SSO, admin bootstrap, tenancy seeder |
✅ | — | ✅ |
| Events: ClickHouse, ingest, batcher, SSE hubs, alert evaluator, issues | — | ✅ | ✅ |
MariaDB, API-key cache, /health + /ready |
✅ | ✅ | ✅ |
- Unset means
both, which is exactly what the service did before roles existed — so deploying this changes nothing until someone opts in. - A route the current role cannot serve is not registered, so it answers
404instead of failing on a dependency the process does not have. UnderMON_ROLE=appthere is no ClickHouse connection at all. - A typo stops the boot.
MON_ROLE=zonris refused by name rather than quietly defaulting — a silent fallback would run the control plane on a host meant to be a zone. - The resolved role is logged once at startup and reported by
GET /health. - Two zones run today.
trailblazeisMON_ROLE=both(it is the control plane and its own zone);applebyisMON_ROLE=zone. What is still not built is cross-zone federation — the control plane can probe a zone's/healthand nothing else, so a zone's projects, keys, alert rules and dashboards are managed by talking to that zone directly. Standing up a zone additionally requires:MON_ZONE_SLUGandMON_PUBLIC_URLset explicitly, aMON_JWT_SIGNING_KEYbyte-identical to the control plane's, its ownMONITOR_API_KEY, and a manualGRANT ALL PRIVILEGES ON monitor.*before the first boot.
Full per-subsystem breakdown in AGENTS.md §6 Roles — the two planes.
An event's project is derived server-side at ingest from the authenticating API key and
overwritten, never trusted from the client. Reads mirror that: QueryAuthMiddleware
resolves a project on every accepted branch, and every read of monitor.events ANDs a
mandatory predicate built by scope.ProjectPredicate — the event and label queries, all
five analytics builders, both issue-event lookups, and the SSE live tail. A context with no
project yields an error, not an unscoped query, so a route registered outside the
middleware fails loudly instead of serving every project.
Where that project comes from differs by credential, and the difference is deliberate.
An API key — ingest or admin — gets it from its own api_keys row, which is a real,
unforgeable tenancy boundary; an admin key reads only its own project, because "admin" is a
scope over verbs, not over tenants. A logged-in session instead selects one with
?project=<slug>, validated against the registry and defaulting to MON_DEFAULT_PROJECT.
That selection is a chooser, not a boundary: Monitor has no per-user project membership
table, so there is nothing that could constrain it — roles (admin/editor/viewer/
pending) gate verbs, which is the authorisation that does exist. Sentry and Grafana work
the same way. A query parameter rather than a header because EventSource cannot send
headers and both SSE streams need the same mechanism; an unknown or retired slug is refused
with a 400 naming it rather than silently falling back, which would show one project's
events under the label the user asked for. AGENTS.md §6 has the full register, including
what changes when a membership table lands.
GET /v1/zones and GET /v1/zones/{zone}/projects list the registry so a project switcher
can populate. Both are reads, available to any authenticated session, and neither should be
called with ?project=, so a stale selection never blocks discovering a valid one. Both hide
retired rows unless asked with ?include_deleted=true — which the admin registry page does,
because a spent slug that is simply missing from the list looks free.
Writing the registry is a different privilege and a different plane. The write surface
lives under /admin, is registered only in a role that runs the control plane, and sits
behind RequireAdmin:
| Route | What it does |
|---|---|
POST /admin/zones |
record a zone — both URLs required |
PUT /admin/zones/{id} |
edit display_name / the URLs; a slug or status in the body is refused, not ignored. Changing query_url discards the stored probe verdict (back to unknown) — it measured the old address |
POST /admin/zones/{id}/retire |
soft-delete; 409 while the zone still owns active projects |
POST /admin/zones/{id}/probe |
probe the zone's query_url now and persist the verdict |
POST /admin/zones/{id}/projects |
create a project inside a zone |
PUT /admin/projects/{id} |
edit display_name; slug, zone_id and status are refused |
POST /admin/projects/{id}/retire |
soft-delete |
Three things about that table are load-bearing. A zone row records infrastructure; it
does not create any — the stack, its ClickHouse, its DNS and its certificate are all
provisioned by hand first, and a row whose query_url points at the wrong box is
syntactically perfect and renders another tenant's data under this zone's name. There is
no DELETE anywhere on it: retiring keeps the row forever so the UNIQUE key on slug
makes a recycled slug impossible, which is what stops a month of surviving events (30-day
TTL) and a permanent daily rollup reattaching to the next owner. And slugs are immutable
at the API, not merely in the UI — a rename that answered 200 and changed nothing would be
invisible here and permanent everywhere else.
POST /admin/zones/{id}/probe compares what the far end says it is (zone + role on its
/health) against what the registry expected, and stores one of
unknown / unconfigured / unreachable / unverified / mismatched / degraded / healthy.
An unreachable zone is a 200 — the probe succeeded and the zone is what is broken —
and mismatched is the verdict that pays for the whole mechanism: a 200-only check would
pass a query_url aimed at another zone and mislabel every read made through it.
A verdict belongs to the URL it measured, so editing query_url throws it away rather
than carrying it forward: query.UpdateZone resets reachability, reachability_detail,
reported_zone and last_probe_at whenever the address actually changes. Keeping them
would leave the row wearing a green healthy earned by a box it no longer points at —
worse than never probing, because unknown admits what it does not know. It lives in the
query layer rather than in the handler so it holds for every caller, not just a UI that
remembers to re-probe.
Transition, expiring 2026-10-06 at the earliest. Rows written before
migrations/006_events_project.sqlread back with an empty project, and the fill is a manual ClickHouse mutation (migrations/manual/backfill_events_project.sql). Until it has run and the 30-day TTL has recycled the rest, the default project also matches those untagged rows; every other project matches exactly. Both arms are dated inscope/scope.goand removed together.
Issues are scoped the same way. monitor.issues lives in MariaDB and never meets
scope.ProjectPredicate, so query.scopeIssues is its counterpart: the list, the count,
every id-addressed read and the update's own WHERE all carry the caller's project. An
issue row is not metadata — it holds the error message taken off the event — so the
listing is scoped for the same reason the events behind it are.
The alerting configuration is scoped the same way again. Migrations 127-130 gave
alert_rules, notification_channels, notification_policies and service_groups a
project, and migrations/007_alert_tables.sql gave the two ClickHouse alerting tables one
— so the rule list, the channel list (which holds credentials), the routing policies, the
alert history and the alert SSE stream are all per project. The one deliberate exception is
query.ListEnabledAlertRules, which the evaluator uses and which must span every project.
Closed gap: timer-driven alert evaluation used to be zone-wide — Evaluator.Run has no
request and therefore no project, so a rule's threshold counted the whole instance while the
HTTP test endpoint (POST /v1/alert-rules/{id}/test) reported the caller's project alone.
With alert_rules.project the evaluator stamps each rule's own project per evaluation, and
the unscoped fall-through is deleted, so the two paths now build an identical statement.
Details in AGENTS.md §6.
Procedures that touch live data or the registry — each irreversible or hard to unpick, and each written because doing it from memory went badly at least once:
| Runbook | |
|---|---|
| Register a zone's project | |
Backfill events.project |
1.69M rows; verification is not a row count |
| Trailblaze's empty registry row | currently blocked, and why |
| Standing up a new zone | the eleven steps, with the undocumented ones marked |
Before any of them, monitor-core check-env runs the same preflight checks the
boot enforces — from the same code path, so the two cannot disagree — and works
against a stack that does not exist yet.
- Request body size: 10 MB for ingestion, 1 MB for analytics queries
- Time series query: Max 90 days range, max 10,000 data points
- Analytics query: Max 10,000 results, max 10 group by fields
- Top N query: Max 1,000 results
- ClickHouse per-query memory: 2 GiB (
CLICKHOUSE_MAX_MEMORY_USAGE) — over it, that one query fails withMEMORY_LIMIT_EXCEEDED(code 241) and nothing else is affected - ClickHouse connection retry: 10 attempts with linear backoff (1s, 2s, ... 10s)
Use the dev CLI for common tasks:
dev help # List available commands
dev up # Start local ClickHouse + MariaDB
dev run # Run the app (auto-migrates both schemas at startup)
dev check # Format, vet, and test
dev down # Stop the local stack
# One-time cutovers (argv subcommands — they migrate, copy, then exit)
go run . backfill-issues # legacy ClickHouse issues → monitor.issues
go run . backfill-config # legacy ClickHouse alerting/dashboard config → MariaDB.
# Must run BEFORE the new binary serves — the server REFUSES
# TO BOOT until it has. See AGENTS.md §8monitor-core/
main.go # Entry: config, MON_ROLE validation, DB connects + migrations, bootstrap, sso.Install(), goroutines
router.go # buildRouter(role) — the HTTP surface, each route gated on the plane that can serve it
Devfile.yaml # Dev CLI commands
Dockerfile # Multi-stage production build (builds the package, `go build .`)
docker-compose.yml # Production stack: monitor-core + ClickHouse + MariaDB (+ monitor-web)
docker-compose.dev.yml # Local development stack
db/
clickhouse.go # ClickHouse connection and batch writer (events)
observe.go # ClickHouse chokepoint: query kind + slow-query warning on every call; errors safe to report
sql.go # MariaDB connection (db.SQL), db.Queryable, db.RunMigrations
migrations/ # MariaDB DDL: identity + tenancy + issues + alert/dashboard config (100_users … 133_service_repos_project)
env/env.go # Environment configuration + RequireProductionSecrets guard
env/role.go # MON_ROLE: the Role type (app/zone/both) and the fail-fast RequireValidRole guard
jwt/jwt.go # Monitor-owned HS512 access/refresh JWTs (alg-pinned)
tools/ # Password.tool.go (bcrypt 12), Crypto.go (AES-256-GCM), Validate.tool.go (+ NormalizeEndpointURL), Slug.tool.go
probe/zone.go # Zone reachability probe: calls a zone's /health + /ready and reports healthy/degraded/unverified/MISMATCHED — the check that catches a registry row pointed at the wrong box
bootstrap/ # First-run seeding: admin.go (first admin user), registry.go (zone + default project)
sso/ # Pluggable SSO wiring onto go-oidc-session: config.go, resolve.go, statestore.go, session.go (Manager)
query/ # MariaDB query layer (squirrel): users/identities/refresh_tokens/sso_*/api_keys/zones/projects/issues/service_repos/alert_rules/notification_channels/notification_policies/service_groups/dashboards/saved_views
scope/ # The request's project: context plumbing + the predicate every event read must carry
structs/ # User/Identity/SSOProvider/SSOSession/RefreshToken/APIKey/Zone/Project/Issue/AlertRule/NotificationChannel/NotificationPolicy/ServiceGroup/Dashboard/SavedView + event/analytics + columns.go (shared identifier regex + column allowlists)
middleware/
session.go # Monitor session auth + Protected/RequireAdmin/RequireEditor/RejectPending
csrf.go # Double-submit CSRF (mon-csrf ↔ X-CSRF-Token)
ingest_auth.go # X-Api-Key auth for POST /v1/events (env master key OR ingest-scope key)
query_auth.go # X-Api-Key (admin) OR Monitor session for /v1/* reads; injects the project (credential-derived for keys, a validated ?project selector for sessions)
logging.go header.go # One http.request.end event per request + failure recorder + panic recovery (SSE-safe); Server header
timeout.go # 25s request-context deadline on /v1 (SSE streams skipped)
gzip.go # gzip for /v1 JSON responses >= 1 KB (SSE, HEAD, POST /v1/api-keys skipped)
responder/responder.go # Standardized JSON response utilities (hands a failure's cause to the request event)
telemetry/ # Self-telemetry to the appleby zone — the only importer of go-monitor (see Self-telemetry)
routes/ # HTTP handlers (thin) — auth + SSO + events/query/analytics/… (see routes/AGENTS.md)
services/ # queue.go batcher.go hub.go query.go analytics.go (ingestion + query engines)
registry/ # Tenancy-registry cache (zone + active projects, 30s refresher) — validates a session's ?project selector
apikeys/ alerts/ issues/ # Subsystems (each Init()s from main.go)
dashboards/ views/ # EMPTY — their tables moved to MariaDB (123/124); the files are tombstones pointing at query/ and structs/
cutover/config.go # One-time ClickHouse → MariaDB copy of the six config tables (`monitor-core backfill-config`)
cutover/guard.go # Refuses the boot while that copy is outstanding — a mis-ordered cutover would otherwise be silent
migrations/
embed.go # In-app ClickHouse migration runner (//go:embed *.sql), rewrites the database name
001_schema.sql 002_add_user_id.sql 003_api_keys.sql 004_events_issue_id.sql
005_issue_occurrences_daily.sql 006_events_project.sql
manual/ # One-off reconciliation SQL — not embedded, not rewritten, run by hand
SELECT * FROM monitor.events LIMIT 10;
-- Find events by trace
SELECT * FROM monitor.events WHERE trace_id = 'trc_z';
-- Find events by service and time range
SELECT * FROM monitor.events
WHERE service = 'users'
AND timestamp >= '2026-02-06 00:00:00';