Skip to content

Fix excessive restarts rule to only monitor elastic-agent process - #17237

Merged
MichelLosier merged 4 commits into
elastic:mainfrom
MichelLosier:fix-elastic-agent-excessive-restarts-rule
Feb 18, 2026
Merged

MichelLosier merged 4 commits into
elastic:mainfrom
MichelLosier:fix-elastic-agent-excessive-restarts-rule

Conversation

@MichelLosier

@MichelLosier MichelLosier commented Feb 4, 2026 •

Copy link
Copy Markdown
Contributor

Proposed commit message

Fixes the excessive restarts rule for elastic_agent package to only monitor the elastic-agent process, and hardcoding of default namespace in alerting rules.

This rescopes the excessive restarts rule to only focus on elastic-agent process restarts. Monitoring other elastic-agent processes could produce false signals since these can occur on config changes.

This also switches the rule to querying logs from the elastic-agent to avoid the existing cross-dependency with the system package:

FROM logs-elastic_agent*
| WHERE message like "Starting*elastic-agent*"
| STATS restart_count = count(*) BY host.name, bucket(@timestamp,5 minute)
| WHERE restart_count > 10
| STATS MAX(restart_count) BY host.name

Possible risk / questions here:

  • The log message could change, or other new log messages could later be introduced with a similar pattern. For elastic agent maintainers: do we expect much risk with this?

An alternative solution using the system package:

FROM metrics-system*
| WHERE TO_LOWER(process.name) == "elastic-agent" AND agent.name NOT LIKE "*agentless*"
| STATS restart_count = COUNT_DISTINCT(process.cpu.start_time) BY host.name, bucket(@timestamp,5 minute)
| WHERE restart_count > 10
| STATS MAX(restart_count) BY host.name

This also addresses an issue where the default namespace was hardcoded into the index pattern of some alerting rules in this package.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@MichelLosier
MichelLosier requested a review from a team as a code owner February 4, 2026 23:01
@MichelLosier MichelLosier added the bug Something isn't working, use only for issues label Feb 4, 2026
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@pierrehilbert
pierrehilbert requested a review from rdner February 5, 2026 07:52
@pierrehilbert pierrehilbert added the Team:Elastic-Agent-Data-Plane Agent Data Plane team [elastic/elastic-agent-data-plane] label Feb 5, 2026
@elasticmachine

Copy link
Copy Markdown

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@andrewkroh andrewkroh added Integration:elastic_agent Elastic Agent Team:Elastic-Agent Platform - Ingest - Agent [elastic/elastic-agent] labels Feb 5, 2026

@belimawr belimawr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the possibility of the log message changing, I don't see it as an issue, but I'd like the @elastic/elastic-agent-data-plane opinion on it.

@AndersonQ

Copy link
Copy Markdown
Contributor

Regarding the possibility of the log message changing, I don't see it as an issue, but I'd like the @elastic/elastic-agent-data-plane opinion on it.

I think you meant @elastic/elastic-agent-control-plane.

While it seems fragile to rely on log messages, they don't change often. But perhaps we can add a comment on the agent to say this message is used here? Worst case we'd need to add a few OR if the log changes, so it should be fine.

I don't recall, but could we get this information from the fleet-agents index? I believe it'd be more reliable getting it from there.

@swiatekm

swiatekm commented Feb 9, 2026

Copy link
Copy Markdown
Member

I'm not against relying on the log file for now, but we should definitely add a test on the agent side which verifies we emit exactly this line and links back to the package definition.

In the long run, it shouldn't be too difficult to have agent expose an uptime metric which would be much more reliable.

@MichelLosier

Copy link
Copy Markdown
Contributor Author

I don't recall, but could we get this information from the fleet-agents index? I believe it'd be more reliable getting it from there.

We do have a .fleet-agents index which has docs for each agent reflecting their current state. Since it is a system index, it has restricted access constraints, and since its just a doc per agent we couldn't do a timeseries calculation of it.

In the long run, it shouldn't be too difficult to have agent expose an uptime metric which would be much more reliable.

As discussed a little bit here: https://github.com/elastic/ingest-dev/issues/6696
I'd like to move some of the agent cpu and memory usage rule queries away from system package data streams to metrics that the elastic-agent can provide. I can create issues for those and this suggestion here.

@MichelLosier

Copy link
Copy Markdown
Contributor Author

Just added in a separate small fix here: 12b0bb8 where the default namespace was hardcoded in the index pattern for some rules

@MichelLosier
MichelLosier force-pushed the fix-elastic-agent-excessive-restarts-rule branch from 12b0bb8 to 9775cd8 Compare February 12, 2026 22:58
@elasticmachine

Copy link
Copy Markdown

💚 Build Succeeded

History

@MichelLosier
MichelLosier merged commit 0f9bf69 into elastic:main Feb 18, 2026
9 checks passed
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package elastic_agent - 2.7.2 containing this change is available at https://epr.elastic.co/package/elastic_agent/2.7.2/

navnit-elastic pushed a commit to navnit-elastic/integrations that referenced this pull request Mar 2, 2026
…astic#17237)

* Fix excessive restarts rule to only monitor elastic-agent process
* Fix hardcoding of default namespace in alerting rules
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working, use only for issues Integration:elastic_agent Elastic Agent Team:Elastic-Agent Platform - Ingest - Agent [elastic/elastic-agent] Team:Elastic-Agent-Data-Plane Agent Data Plane team [elastic/elastic-agent-data-plane]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants

⌂ 换网址