Alerting on cron and background jobs that never ran, hung, or did nothing (in 6 languages)

Our HTTP alerting was in good shape. 500s, slow endpoints, error rates: all covered. Yet the incidents that hurt most kept coming from somewhere else:

• The 02:00 invoice job never started, because the server restarted and the scheduler got stuck.
• A queue consumer was up, but processed nothing for 20 minutes.
• The bank statement file didn't arrive, so the import "succeeded" with 0 rows.
• A worker loop deadlocked, and its exception was swallowed in a catch.

None of these has an HTTP status code. Ping-based monitors (Healthchecks.io, Cronitor and friends) cover part of it, but they add another SaaS, a ping URL in every job, and they can't tell "succeeded" from "succeeded with nothing done".

So I built RunSignal: cron and background-job monitoring on OpenTelemetry, with alerts as code, for .NET, Java, Go, Node.js, Python and Rust.

👉 GitHub: github.com/gokhanertogan/RunSignal (MIT)

How it works

1. Record. An SDK records each run of a workload (success, failure, cancellation, duration), the items it handled, and heartbeats from long-running loops, as OpenTelemetry metrics. Every SDK uses the same metric names.
2. Describe. Each workload declares what it should do: a cron schedule, a max duration, a heartbeat interval, a minimum output, a failure rate. The running app writes these to a JSON manifest.
3. Provision. In CI, runsignal provision turns the manifest into alert conditions and a dashboard in New Relic, or runsignal export prometheus writes Prometheus alerting rules (plus a Grafana dashboard).

Alerts are evaluated by the backend, never in your process, because a process that has died cannot report that it is missing. Because the metrics are identical in every language, one CLI provisions alerts for a whole polyglot system.

The same idea in six languages

Where a scheduler exists, there is no code in the jobs: the integration records every run and registers the job with its cron at startup, so a job that never fires still gets a missed alert.

.NET: Hangfire (Quartz.NET works the same way):

builder.Services.AddRunSignal().AddHangfire();
builder.Services.AddHangfire((sp, c) => c.UseSqlServerStorage(cs).UseRunSignal(sp));

[Workload("nightly-invoice", MaxDuration = "30m", MinProcessed = 100)]
public async Task GenerateInvoices(PerformContext? context = null) { /* ... */ }

Java: Spring @Scheduled (and Spring Batch), through Spring's own observations, with no proxies:

@Scheduled(cron = "0 0 2 * * *")
@RunSignal(value = "nightly-invoice", maxDuration = "30m", minProcessed = 100)
void generateInvoices() { ... }

Python: Celery tasks and beat schedules (APScheduler too), or a plain with block:

from runsignal import default_registry

invoices = default_registry.workload("nightly-invoice", schedule="0 2 * * *", max_duration="30m", min_processed=100)

with invoices.run():
    for order in unbilled_orders():
        create_invoice(order)
        invoices.processed()

Node.js: node-cron, croner, cron or node-schedule (BullMQ too):

import { cronJob } from '@runsignal/node/cron';

const invoices = cronJob('nightly-invoice', { schedule: '0 2 * * *', maxDuration: '30m' }, async (w) => {
  for (const order of await unbilledOrders()) { await createInvoice(order); w.processed(); }
});
cron.schedule(invoices.schedule, invoices.run);

Go: robfig/cron, or Run with context cancellation and panic handling:

rscron.AddFunc(c, runsignal.Default, "nightly-invoice", "0 2 * * *",
    func(ctx context.Context) error { return generateInvoices(ctx) },
    runsignal.MaxDuration(30*time.Minute))

Rust: sync and async runs; a dropped future is a cancelled run:

let invoices = registry.workload("nightly-invoice", Options::new()
    .schedule("0 2 * * *")
    .max_duration(Duration::from_secs(30 * 60)))?;

invoices.run(|| generate_invoices())?;

The alerts

hung is new in 0.3.0. A duration histogram only sees runs that finish, so a run stuck forever never shows up there. Every SDK now reports the age of the oldest unfinished run as a gauge, so the alert fires while the run is still hanging.

Jobs with no SDK at all

Since 0.3.0, runsignal exec wraps any command (a shell script, a crontab entry, a container in a Kubernetes CronJob) as one monitored run:

runsignal exec --workload nightly-export --service billing -- ./export.sh

• Outcome: exit code 0 is a success, anything else a failure with its exit code.
• Signals: stop signals are forwarded and count as cancelled, and the command's exit code passes through.
• Packaging: it's a 10 MB native executable with no runtime dependency. Copy it into any image with COPY --from=ghcr.io/gokhanertogan/runsignal.
• Kubernetes: runsignal discover kubernetes writes the manifest straight from your CronJobs, including schedules, time zones and activeDeadlineSeconds.

Design choices

• OpenTelemetry-native. These are plain OTel instruments, so the data goes to any OTLP backend. Alert provisioning targets New Relic and Prometheus today.
• One contract, six SDKs. A shared conformance test defines the exact metrics a scenario must produce, and every SDK runs it in CI. An SDK that drifts fails its own build.
• Cheap on the hot path. Recording a run doesn't allocate in .NET and Go (enforced by tests), and costs well under a microsecond in Rust and Node.
• Idempotent provisioning. provision changes only what changed and never touches conditions it didn't create. Run it on every deploy.

Status

It's MIT-licensed and pre-1.0. Every alert kind has fired on a real New Relic account, and the Prometheus rules are covered by promtool test rules.

• GitHub: github.com/gokhanertogan/RunSignal
• Packages:
  • NuGet
  • Maven Central
  • PyPI
  • npm
  • Go
  • crates.io