Skip to content

Add cluster UUID to the Logstash node datastream. - #9971

Merged
mashhurs merged 5 commits into
elastic:mainfrom
mashhurs:add-cluster-id-to-logstash-node-datastream
May 27, 2024
Merged

mashhurs merged 5 commits into
elastic:mainfrom
mashhurs:add-cluster-id-to-logstash-node-datastream

Conversation

@mashhurs

@mashhurs mashhurs commented May 24, 2024 •

Copy link
Copy Markdown
Contributor

Proposed commit message

Logstash monitoring with metricbeat and self include cluster_uuid which will be used during the queries (example telemetry collection). However, with agent driven monitoring we are losing the cluster UUID. This change fixes the explained bug.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Use the Logstash integration, install the agent and copy the change script into the policy. We can see the cluster UUID will be reflected in .ds-metrics-logstash.node* indices. Also query against cluster_uuid (which is an alias for logstash.elasticsearch.cluster.id) also works as expected.

// example query
GET .ds-metrics-logstash.node*/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "terms": {
            "cluster_uuid": [
              "wjK4bDWFQxKggvldnpKeng"
            ]
          }
        },
        {
          "bool": {
            "should": [
              {
                "term": {
                  "type": "logstash_stats"
                }
              },
              {
                "term": {
                  "metricset.name": "node_stats"
                }
              },
              {
                "term": {
                  "data_stream.dataset": "logstash.node"
                }
              }
            ]
          }
        },
        {
          "range": {
            "@timestamp": {
              "format": "epoch_millis",
              "gte": 0,
              "lte": 2816489716793
            }
          }
        }
      ]
    }
  },
  "collapse": {
    "field": "host.id"
  },
  "sort": [
    {
      "@timestamp": {
        "order": "desc",
        "unmapped_type": "long"
      }
    }
  ],
  "from": 0,
  "size": 10000
}

Related issues

Screenshots

@mashhurs mashhurs added bug Something isn't working, use only for issues bugfix Pull request that fixes a bug issue labels May 24, 2024
@mashhurs mashhurs self-assigned this May 24, 2024
@mashhurs
mashhurs requested a review from a team as a code owner May 24, 2024 07:23
@mashhurs mashhurs linked an issue May 24, 2024 that may be closed by this pull request
@mashhurs
mashhurs requested a review from robbavey May 24, 2024 07:31
@klacabane

klacabane commented May 24, 2024 •

Copy link
Copy Markdown
Contributor

I've started a stack and the logstash service defined in the _dev/deploy directory. after configuring the integration to collect metrics from the previously started logstash I can't see the cluster_uuid populated in the logstash.node documents. If I'm interpreting the code correctly we should extract that property from /_node/stats?graph=true vertices but I don't see it when I hit the endpoint manually.
Is there any configuration missing (maybe in logstash.yml) for this property to be output in the /_node/stats response ?

@mashhurs

Copy link
Copy Markdown
Contributor Author

I've started a stack and the logstash service defined in the _dev/deploy directory. after configuring the integration to collect metrics from the previously started logstash I can't see the cluster_uuid populated in the logstash.node documents. If I'm interpreting the code correctly we should extract that property from /_node/stats?graph=true vertices but I don't see it when I hit the endpoint manually. Is there any configuration missing (maybe in logstash.yml) for this property to be output in the /_node/stats response ?

Thanks @klacabane !
I failed to edit the URL 🤦 now it is correctly reflected.

Map with a proper data shape.

Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
@elasticmachine

Copy link
Copy Markdown

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@elasticmachine

Copy link
Copy Markdown

💚 Build Succeeded

History

cc @mashhurs

@elastic-sonarqube

Copy link
Copy Markdown

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

@mashhurs
mashhurs requested review from klacabane and robbavey May 24, 2024 18:01

@robbavey robbavey left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@klacabane klacabane left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mashhurs
mashhurs merged commit a45ab59 into elastic:main May 27, 2024
@elasticmachine

Copy link
Copy Markdown

Package logstash - 2.4.7 containing this change is available at https://epr.elastic.co/search?package=logstash

@mashhurs
mashhurs deleted the add-cluster-id-to-logstash-node-datastream branch February 10, 2025 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working, use only for issues bugfix Pull request that fixes a bug issue Integration:logstash Logstash

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Logstash integration] Add cluster_uuid to node datastream.

5 participants

⌂ 换网址