Skip to content

github: add teams data stream and link maintainers by entity id - #22121

Merged
muskan-agarwal26 merged 6 commits into
elastic:mainfrom
muskan-agarwal26:github-team-datastream
Oct 29, 2026
Merged

muskan-agarwal26 merged 6 commits into
elastic:mainfrom
muskan-agarwal26:github-team-datastream

Conversation

@muskan-agarwal26

@muskan-agarwal26 muskan-agarwal26 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Proposed commit message

The members data stream recorded teams a user maintains under
user.entity.relationships.administers.service.*. The Entity Store kept those
values as raw identifiers and never created a team entity, so the Administers
relationship could not be joined.

Maintainers now set user.entity.relationships.administers.entity.id to the
team GraphQL node id. The previous administers.service.id and
administers.service.name fields are removed. A new github.teams data stream
emits one asset document per organization team, with that same node id in
entity.id, and no user.* fields. github.teams.combined_slug keeps org/team-slug.
The team url is copied to entity.reference. createdAt and updatedAt are stored
as github.teams.created_at and github.teams.updated_at.

Each collection stores a new team
document so entity.last_seen_timestamp advances. The member document id changes
once in 3.0.0 so an unchanged member is indexed again with the new fields.

The Organization Members dashboard terms panel uses
github.members.maintained_team_names. The unique-count panel uses
user.entity.relationships.administers.entity.id. An Organization Teams
dashboard counts distinct teams by entity.id.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

From packages/github, with the Elastic stack up:

elastic-package test

Related issues

Screenshots

github-members-dashboard

@muskan-agarwal26
muskan-agarwal26 requested review from a team as code owners October 22, 2026 11:32
@muskan-agarwal26 muskan-agarwal26 self-assigned this Oct 22, 2026
@muskan-agarwal26 muskan-agarwal26 added breaking change enhancement New feature or request Integration:github GitHub Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] labels Oct 22, 2026
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@github-actions

Copy link
Copy Markdown
Contributor

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!


The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

- set:
field: entity.type
tag: set_entity_type
value: Team

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟡 Medium confidence: medium path: packages/github/data_stream/teams/elasticsearch/ingest_pipeline/default.yml:82

The teams pipeline sets entity.type to the capitalised value Team and redeclares the ECS field in fields/fields.yml. Use a lowercase value and declare the field with external: ecs.

Details

set_entity_type writes Team. The entity guidance requires lowercase entity.type values, and the members stream in this package already writes lowercase user.entity.type: [user] (members default.yml line 117). A query or join on entity.type: team will not match a keyword value of Team. The PR description says the value is declared locally because Team is not in the ECS set. That local declaration doesn't change anything: entity.type is a keyword in ECS either way, and ECS does not validate its values. All the custom definition in data_stream/teams/fields/fields.yml does is override an ECS field package-locally. Every other entity.* field in this stream is declared external: ecs in fields/ecs.yml. The GitHub-specific classification already lives in entity.sub_type: GitHub Team. Note that no value in the entity catalog's allowed table clearly fits a team, so team is also outside that table. The capitalisation and the local override are the concrete defects here.

Recommendation:

Write a lowercase type and keep the GitHub detail in entity.sub_type. Then move the declaration to fields/ecs.yml and delete the custom entity.type entry from fields/fields.yml.

  - set:
      field: entity.type
      tag: set_entity_type
      value:
        - team
# data_stream/teams/fields/ecs.yml
- name: entity.type
  external: ecs

Before settling on a value, check which entity.type the Entity Store expects for group-like generic entities.

Also in: packages/github/data_stream/teams/fields/fields.yml:1


🤖 AI-Generated Review | Vera Review Bot - v0.4.3 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@muskan-agarwal26 can you check this? Seems valid.

cc: @chemamartinez

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The entity type field description says:

Important: The field value must be one of the following:

bucket, database, container, function, queue, host, user, application, service, session, cloud, orchestrator

so I am not sure if we should add a different value here. If it the team entity doesn't fit in any of the allowed values I think we should remove the field at all.

@alexreal1314 do you have any preference on this?

},
"scale": "ordinal",
"sourceField": "user.entity.relationships.administers.service.name"
"sourceField": "user.entity.relationships.administers.entity.id"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🔵 Low confidence: high path: packages/github/kibana/dashboard/github-2398d0ec-abdd-410c-b5e9-c43d0ab7330a.json:1093

The "Teams by maintainer count" panel now buckets on opaque GraphQL team node IDs instead of team names. Expose a readable maintained-team-name field and use it for the terms axis.

Details

This terms column used to read user.entity.relationships.administers.service.name and was labelled Team. It now reads user.entity.relationships.administers.entity.id and is labelled Team ID, so every bar shows a node ID such as T_kgDO.... The members pipeline no longer writes any maintained-team name. The build_administers_entity_ids script (members default.yml lines 142-163) collects only IDs. github.members.teams is declared as a plain group, not nested, so Lens cannot narrow it to MAINTAINER entries. The other changed column (line 1209) is a unique count, so the switch to IDs doesn't hurt it.

Recommendation:

Have the existing script also collect maintained team names into a custom field. Declare that field and use it as this panel's terms sourceField.

def adminIds = [];
def adminNames = [];
for (def team : ctx.json.teams) {
  if (team?.role == "MAINTAINER" && team?.id != null) {
    adminIds.add(team.id);
    if (team?.name != null) { adminNames.add(team.name); }
  }
}
// existing adminIds handling unchanged
if (!adminNames.isEmpty()) {
  ctx.github = ctx.github ?: [:];
  ctx.github.members = ctx.github.members ?: [:];
  ctx.github.members.maintained_team_names = adminNames;
}
# data_stream/members/fields/fields.yml, under github.members
- name: maintained_team_names
  type: keyword
  description: Display names of teams where the member is a MAINTAINER.

🤖 AI-Generated Review | Vera Review Bot - v0.4.3 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

owner: example-org
interval: 2s
# Two identical snapshots of 3 teams. The ingest fingerprint of event.original
# reuses _id, so the second snapshot overwrites the first and the index holds 3

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🔵 Low confidence: high path: packages/github/data_stream/teams/_dev/test/system/test-default-config.yml:12

The system test comment says the second identical snapshot overwrites the first. Data streams index with op_type create, so the duplicate _id is rejected and the first document is kept. Correct the comment.

Details

The teams pipeline sets _id from a fingerprint of event.original (default.yml line 146). Data streams accept only create operations, so a later identical snapshot gets a version conflict in the same backing index rather than replacing the earlier document. The hit_count: 3 assertion still holds, but the stated mechanism is wrong. One practical consequence: for an unchanged team, @​timestamp and entity.last_seen_timestamp keep the value from the first indexed snapshot until the backing index rolls over. The members stream uses the same fingerprint pattern (members default.yml line 351), so this follows an existing package convention. Whether the Entity Store's lookback tolerates it is not something the source can settle.

Recommendation:

Rewrite the comment so it describes the real behaviour (duplicates are rejected and the first document is kept). If teams need entity.last_seen_timestamp to advance every cycle, drop the _id fingerprint so each collection cycle indexes a fresh asset document.


🤖 AI-Generated Review | Vera Review Bot - v0.4.3 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

@qcorporation qcorporation added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Oct 22, 2026
Copilot AI balanced review requested due to automatic review settings October 26, 2026 10:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Content-only fingerprints suppress recurring team observations and can prevent unchanged members from receiving the migrated relationship fields after upgrade.

2 open issues
What changed in this PR

This PR adds GitHub teams as generic entities and links maintainers to them using matching GitHub node IDs, enabling Administers relationships in the Entity Store.

Changes:

  • Adds a paginated teams collector, entity pipeline, field mappings, and test fixtures.
  • Migrates member relationships from service fields to entity IDs while preserving team display names for dashboards.
  • Updates documentation and releases the breaking schema change as version 3.0.0.
File Description
packages/​github/​manifest.yml Bumps version and updates collection descriptions.
packages/​github/​kibana/​dashboard/​github-2398d0ec-abdd-410c-b5e9-c43d0ab7330a.json Uses maintained team names and entity IDs.
packages/​github/​docs/​README.md Documents teams and migrated member relationships.
packages/​github/​data_stream/​teams/​sample_event.json Shows a team entity event.
packages/​github/​data_stream/​teams/​manifest.yml Defines teams collection settings.
packages/​github/​data_stream/​teams/​fields/​fields.yml Defines team attributes and entity classification.
packages/​github/​data_stream/​teams/​fields/​ecs.yml Declares standard entity and event fields.
packages/​github/​data_stream/​teams/​fields/​base-fields.yml Defines dataset metadata.
packages/​github/​data_stream/​teams/​fields/​agent.yml Declares agent-related fields.
packages/​github/​data_stream/​teams/​elasticsearch/​ingest_pipeline/​default.yml Maps team snapshots to generic entities.
packages/​github/​data_stream/​teams/​agent/​stream/​cel.yml.hbs Collects paginated GraphQL team snapshots.
packages/​github/​data_stream/​teams/​_dev/​test/​system/​test-default-config.yml Configures collection and deduplication assertions.
packages/​github/​data_stream/​teams/​_dev/​test/​pipeline/​test-github-teams.log-expected.json Defines expected team pipeline output.
packages/​github/​data_stream/​teams/​_dev/​test/​pipeline/​test-github-teams.log Supplies team pipeline fixtures.
packages/​github/​data_stream/​teams/​_dev/​test/​pipeline/​test-common-config.yml Sets shared pipeline test metadata.
packages/​github/​data_stream/​members/​sample_event.json Refreshes the member relationship example.
packages/​github/​data_stream/​members/​fields/​fields.yml Adds maintained team display names.
packages/​github/​data_stream/​members/​fields/​ecs.yml Replaces service relationship fields with entity IDs.
packages/​github/​data_stream/​members/​elasticsearch/​ingest_pipeline/​default.yml Emits maintainer entity links and team names.
packages/​github/​data_stream/​members/​_dev/​test/​pipeline/​test-github-members-member-sample.log-expected.json Updates expected member relationships.
packages/​github/​changelog.yml Records the breaking change and teams addition.
packages/​github/​_dev/​deploy/​docker/​files/​config-teams.yml Adds mock GraphQL responses.
packages/​github/​_dev/​deploy/​docker/​docker-compose.yml Registers the teams mock service.
packages/​github/​_dev/​build/​docs/​README.md Updates the documentation source template.

Review effort: Balanced

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

Comment on lines +3 to +11
changes:
- description: |
Remove `user.entity.relationships.administers.service.id` and `user.entity.relationships.administers.service.name` from the members data stream.
Maintainers now set `user.entity.relationships.administers.entity.id` to the GitHub team node id so the Entity Store can join the relationship.
type: breaking-change
link: https://github.com/elastic/integrations/pull/22121
- description: Add a teams data stream that emits one generic entity document per GitHub organization team.
type: enhancement
link: https://github.com/elastic/integrations/pull/22121

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add to commit message (short) why both enhancement + breaking-change should coexist together?

"iam"
],
"kind": "asset",
"original": "{\"combinedSlug\":\"example-org/frontend-team\",\"databaseId\":1002,\"description\":null,\"id\":\"T_kgDOAADDEE\",\"name\":\"Frontend Team\",\"parentTeam\":{\"id\":\"T_kgDOAABBCC\",\"slug\":\"backend-team\"},\"privacy\":\"SECRET\",\"slug\":\"frontend-team\"}",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know its not a lot of fields and primary usecase is elsewhere (Entity Analytics view), but just for quality can you add a minimal dashboard for this data stream?

"Authorization": ["Bearer " + state.access_token],
},
"Body": {
"query": "query OrgTeams($org: String!, $teamCursor: String) { organization(login: $org) { teams(first: 20, after: $teamCursor, orderBy: { field: NAME, direction: ASC }) { pageInfo { hasNextPage endCursor } totalCount edges { node { id databaseId name slug combinedSlug privacy description parentTeam { id slug } } } } } rateLimit { cost remaining resetAt } }",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few Team fields worth adding: url mapped to the ECS entity.reference (currently empty), and createdAt/updatedAt as github.teams.created_at/updated_at.

"message": "POST /graphql (teams): " + ((body.?errors.orValue(null) != null) ? ((body.errors.size() > 0) ? body.errors[0].?message.orValue("GraphQL error") : "GraphQL error") : "missing organization.teams in response"),
},
},
"want_more": false,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error results don't set cursor (here and in the error branch at L87). A failure part-way through therefore leaves cursor.after set, and the next interval resumes mid-roster with a partial snapshot. Add "cursor": {} to both error branches so the next run starts from the beginning.

Copilot AI balanced review requested due to automatic review settings October 28, 2026 09:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Team assets lack the required entity.type classification, and the dashboard counts parent slugs rather than stable team IDs, causing renamed parent teams to be overcounted.

2 open issues
2 resolved since last review

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 28, 2026 09:51
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

✅ All changelog entries have the correct PR link.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The ingest fixtures establish event shapes but do not verify that Entity Store materializes the generic team entity and Entity Graph renders its Administers edge in a live cluster.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@vera-review-bot

Copy link
Copy Markdown
Contributor

🟢 No issues across the latest commits dc7265e.

⚠️ 3 comments still unresolved from earlier commits — 1 medium, 2 low
  • 🟡 The teams pipeline sets entity.type to the capitalised value Team and redeclares the ECS field in fields/fields.yml. Use a lowercase value and declare the field with external: ecs. (link)
  • 🔵 The "Teams by maintainer count" panel now buckets on opaque GraphQL team node IDs instead of team names. Expose a readable maintained-team-name field and use it for the terms axis. (link)
  • 🔵 The system test comment says the second identical snapshot overwrites the first. Data streams index with op_type create, so the duplicate _id is rejected and the first document is kept. Correct the comment. (link)

Review summary

Issues found across earlier commits 0ffb2e2 — 1 medium, 2 low
  • 🟡 The teams pipeline sets entity.type to the capitalised value Team and redeclares the ECS field in fields/fields.yml. Use a lowercase value and declare the field with external: ecs. (link) (Outdated)
  • 🔵 The "Teams by maintainer count" panel now buckets on opaque GraphQL team node IDs instead of team names. Expose a readable maintained-team-name field and use it for the terms axis. (link) (Outdated)
  • 🔵 The system test comment says the second identical snapshot overwrites the first. Data streams index with op_type create, so the duplicate _id is rejected and the first document is kept. Correct the comment. (link) (Outdated)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.4.3 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

💚 Build Succeeded

History

cc @muskan-agarwal26

@chemamartinez chemamartinez left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mergify

mergify Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again.

@kcreddy kcreddy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@muskan-agarwal26
muskan-agarwal26 merged commit 4cb2e93 into elastic:main Oct 29, 2026
17 checks passed
@muskan-agarwal26
muskan-agarwal26 deleted the github-team-datastream branch October 29, 2026 06:03
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package github - 3.0.0 containing this change is available at https://epr.elastic.co/package/github/3.0.0/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking change documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. enhancement New feature or request Integration:github GitHub Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[github] Add a teams data stream so GitHub teams exist as generic entities and link to members via administers

5 participants

⌂ 换网址