Per Service CPU Usage Monitoring - A Job For Prometheus & Grafana

Recently we had a situation where it would have been good to have had per service CPU usage graphed over time.

Some of our servers are run in a 'multi tenant' mode where we deliver a service to 20 different customers, with each customer using their own dedicated systemd service. Some of these customers use the service quite aggressively while others are well behaved.

Recently we restarted one of the services of an 'aggressive' customer in order to realize a new configuration and noticed that the overall CPU usage on the server dropped in half. This observation spawned a lot of questions...

• Was this one service accounting for 50% of the overall CPU usage on the server?
• Was the new configuration more efficient in some way?
• Could it be coincidence that this service restart resulted in less CPU consumption?
• Did the CPU usage for this particular service increase over time, or had it always been like this?

The server had a lot of CPU headroom, there was no immediate danger, yet it felt like there was a tiger in the bushes waiting for us. Suddenly we felt the need to increase our situational awareness.

Our monitoring server graphed total CPU usage, but had no notion of 'per process CPU usage'. If we had a way to visualize each customers CPU utilization it could help answer a lot of questions.

So! How to get this situational awareness? After discussing with our staff, searching the internet, and posing the question to our AI agents, a solution using Prometheus for data collection and Grafana for visualization was selected.

10,000 Foot View

• A prometheus agent runs on the customer facing server that exposes per process related statistics
• Prometheus collector runs on the monitoring server, polls the customer facing server and updates a data store
• Grafana queries the prometheus datastore and renders time series graphs

Multi Tenant Server Provisioning

First we installed prometheus-process-exporter on the multi-tenant server. By default the prometheus-process-exporter service has no configuration. The only configuration related file installed by default is /etc/default/prometheus-process-exporter and looks like this:

# Set the command-line arguments to pass to the server.
# Due to shell escaping, to pass backslashes for regexes, you need to double
# them (\\d for \d). If running under systemd, you need to double them again
# (\\\\d to mean \d), and escape newlines too.
ARGS=""

We need to update this file to listen on a TCP socket and point it to a configuration file. The configuration file uses the yaml format. Here is an example we placed in /etc/prometheus/process-exporter.yml (the /etc/prometheus directory may need to be created):

process_names:
  - name: "{{.Cgroups}}"
    comm:
      - rust-widget

In our case every customer runs the same binary for their service, the binary name being rust-widget. The Cgroups argument will name each group after the seen service name, each systemd service gets its own group and children forked by it are associated with it.

Update the ARGS declaration in /etc/default/prometheus-process-exporter:

ARGS="--config.path=/etc/prometheus/process-exporter.yml --web.listen-address=10.10.10.10:9256"

Then restart the prometheus-process-exporter service to realize the change

Monitoring Server Provisioning

Next we need to configure the monitoring server to poll this new prometheus endpoint. To do this we'll add a 'job' to /etc/prometheus/prometheus.yml. This job goes in the scrape_configs section:

  - job_name: multi-tenant-1
    scrape_interval: 1m
    scrape_timeout: 1m
    static_configs:
      - targets: ['10.10.10.10:9256']
    metric_relabel_configs:
      - source_labels: [groupname]
        regex: '.*/([^/]+rust-widget)\.service.*'
        target_label: service
        replacement: '$1'

The metric_relabel_configs section is worth talking about. In our case all our customer facing services have a naming convention of <customer>_rust-widget. Since we are only interested in CPU usage information of these services we use a regex that matches that.

After changing this file restart prometheus to realize it. We can verify the state of the new endpoint by having a look at the prometheus web server. the default web server port for prometheus is 9090. The path that provides endpoint status is: http://127.0.0.1:9090/classic/targets. If prometheus is able to query the new endpoint and things are working well the endpoint State will be 'UP'.

Now, how to visualize this? Either create or edit an existing dashboard, then add a time series panel. Select the prometheus data source (we made the assumption that prometheus is already configured as a data source for the grafana instance). We used Claude to recommend a ProMQ query as most humans have little interest in learning this query language. Claude returned this for a per job graph and it worked as desired strait away:

sum by (service) (rate(namedprocess_namegroup_cpu_seconds_total{job="multi-tenant-1", service=~".+"}[5m]))

The resulting graph looks like this:

The next time we observe some odd change in CPU consumption we will be better positioned to know why. Reducing the tigers in the bushes one at a time...