Repository navigation
github: add teams data stream and link maintainers by entity id - #22121
Conversation
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
✅ Elastic Docs Style Checker (Vale)No issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
| - set: | ||
| field: entity.type | ||
| tag: set_entity_type | ||
| value: Team |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/github/data_stream/teams/elasticsearch/ingest_pipeline/default.yml:82
The teams pipeline sets entity.type to the capitalised value Team and redeclares the ECS field in fields/fields.yml. Use a lowercase value and declare the field with external: ecs.
Details
set_entity_type writes Team. The entity guidance requires lowercase entity.type values, and the members stream in this package already writes lowercase user.entity.type: [user] (members default.yml line 117). A query or join on entity.type: team will not match a keyword value of Team. The PR description says the value is declared locally because Team is not in the ECS set. That local declaration doesn't change anything: entity.type is a keyword in ECS either way, and ECS does not validate its values. All the custom definition in data_stream/teams/fields/fields.yml does is override an ECS field package-locally. Every other entity.* field in this stream is declared external: ecs in fields/ecs.yml. The GitHub-specific classification already lives in entity.sub_type: GitHub Team. Note that no value in the entity catalog's allowed table clearly fits a team, so team is also outside that table. The capitalisation and the local override are the concrete defects here.
Recommendation:
Write a lowercase type and keep the GitHub detail in entity.sub_type. Then move the declaration to fields/ecs.yml and delete the custom entity.type entry from fields/fields.yml.
- set:
field: entity.type
tag: set_entity_type
value:
- team# data_stream/teams/fields/ecs.yml
- name: entity.type
external: ecsBefore settling on a value, check which entity.type the Entity Store expects for group-like generic entities.
Also in: packages/github/data_stream/teams/fields/fields.yml:1
🤖 AI-Generated Review | Vera Review Bot - v0.4.3 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
@muskan-agarwal26 can you check this? Seems valid.
cc: @chemamartinez
There was a problem hiding this comment.
The entity type field description says:
Important: The field value must be one of the following:
bucket, database, container, function, queue, host, user, application, service, session, cloud, orchestrator
so I am not sure if we should add a different value here. If it the team entity doesn't fit in any of the allowed values I think we should remove the field at all.
@alexreal1314 do you have any preference on this?
| }, | ||
| "scale": "ordinal", | ||
| "sourceField": "user.entity.relationships.administers.service.name" | ||
| "sourceField": "user.entity.relationships.administers.entity.id" |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: high path: packages/github/kibana/dashboard/github-2398d0ec-abdd-410c-b5e9-c43d0ab7330a.json:1093
The "Teams by maintainer count" panel now buckets on opaque GraphQL team node IDs instead of team names. Expose a readable maintained-team-name field and use it for the terms axis.
Details
This terms column used to read user.entity.relationships.administers.service.name and was labelled Team. It now reads user.entity.relationships.administers.entity.id and is labelled Team ID, so every bar shows a node ID such as T_kgDO.... The members pipeline no longer writes any maintained-team name. The build_administers_entity_ids script (members default.yml lines 142-163) collects only IDs. github.members.teams is declared as a plain group, not nested, so Lens cannot narrow it to MAINTAINER entries. The other changed column (line 1209) is a unique count, so the switch to IDs doesn't hurt it.
Recommendation:
Have the existing script also collect maintained team names into a custom field. Declare that field and use it as this panel's terms sourceField.
def adminIds = [];
def adminNames = [];
for (def team : ctx.json.teams) {
if (team?.role == "MAINTAINER" && team?.id != null) {
adminIds.add(team.id);
if (team?.name != null) { adminNames.add(team.name); }
}
}
// existing adminIds handling unchanged
if (!adminNames.isEmpty()) {
ctx.github = ctx.github ?: [:];
ctx.github.members = ctx.github.members ?: [:];
ctx.github.members.maintained_team_names = adminNames;
}
# data_stream/members/fields/fields.yml, under github.members
- name: maintained_team_names
type: keyword
description: Display names of teams where the member is a MAINTAINER.🤖 AI-Generated Review | Vera Review Bot - v0.4.3 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| owner: example-org | ||
| interval: 2s | ||
| # Two identical snapshots of 3 teams. The ingest fingerprint of event.original | ||
| # reuses _id, so the second snapshot overwrites the first and the index holds 3 |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: high path: packages/github/data_stream/teams/_dev/test/system/test-default-config.yml:12
The system test comment says the second identical snapshot overwrites the first. Data streams index with op_type create, so the duplicate _id is rejected and the first document is kept. Correct the comment.
Details
The teams pipeline sets _id from a fingerprint of event.original (default.yml line 146). Data streams accept only create operations, so a later identical snapshot gets a version conflict in the same backing index rather than replacing the earlier document. The hit_count: 3 assertion still holds, but the stated mechanism is wrong. One practical consequence: for an unchanged team, @timestamp and entity.last_seen_timestamp keep the value from the first indexed snapshot until the backing index rolls over. The members stream uses the same fingerprint pattern (members default.yml line 351), so this follows an existing package convention. Whether the Entity Store's lookback tolerates it is not something the source can settle.
Recommendation:
Rewrite the comment so it describes the real behaviour (duplicates are rejected and the first document is kept). If teams need entity.last_seen_timestamp to advance every cycle, drop the _id fingerprint so each collection cycle indexes a fresh asset document.
🤖 AI-Generated Review | Vera Review Bot - v0.4.3 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
…github-team-datastream
There was a problem hiding this comment.
🟡 Changes recommended
Content-only fingerprints suppress recurring team observations and can prevent unchanged members from receiving the migrated relationship fields after upgrade.
2 open issues
What changed in this PR
This PR adds GitHub teams as generic entities and links maintainers to them using matching GitHub node IDs, enabling Administers relationships in the Entity Store.
Changes:
- Adds a paginated teams collector, entity pipeline, field mappings, and test fixtures.
- Migrates member relationships from service fields to entity IDs while preserving team display names for dashboards.
- Updates documentation and releases the breaking schema change as version 3.0.0.
| File | Description |
|---|---|
packages/github/manifest.yml |
Bumps version and updates collection descriptions. |
packages/github/kibana/dashboard/github-2398d0ec-abdd-410c-b5e9-c43d0ab7330a.json |
Uses maintained team names and entity IDs. |
packages/github/docs/README.md |
Documents teams and migrated member relationships. |
packages/github/data_stream/teams/sample_event.json |
Shows a team entity event. |
packages/github/data_stream/teams/manifest.yml |
Defines teams collection settings. |
packages/github/data_stream/teams/fields/fields.yml |
Defines team attributes and entity classification. |
packages/github/data_stream/teams/fields/ecs.yml |
Declares standard entity and event fields. |
packages/github/data_stream/teams/fields/base-fields.yml |
Defines dataset metadata. |
packages/github/data_stream/teams/fields/agent.yml |
Declares agent-related fields. |
packages/github/data_stream/teams/elasticsearch/ingest_pipeline/default.yml |
Maps team snapshots to generic entities. |
packages/github/data_stream/teams/agent/stream/cel.yml.hbs |
Collects paginated GraphQL team snapshots. |
packages/github/data_stream/teams/_dev/test/system/test-default-config.yml |
Configures collection and deduplication assertions. |
packages/github/data_stream/teams/_dev/test/pipeline/test-github-teams.log-expected.json |
Defines expected team pipeline output. |
packages/github/data_stream/teams/_dev/test/pipeline/test-github-teams.log |
Supplies team pipeline fixtures. |
packages/github/data_stream/teams/_dev/test/pipeline/test-common-config.yml |
Sets shared pipeline test metadata. |
packages/github/data_stream/members/sample_event.json |
Refreshes the member relationship example. |
packages/github/data_stream/members/fields/fields.yml |
Adds maintained team display names. |
packages/github/data_stream/members/fields/ecs.yml |
Replaces service relationship fields with entity IDs. |
packages/github/data_stream/members/elasticsearch/ingest_pipeline/default.yml |
Emits maintainer entity links and team names. |
packages/github/data_stream/members/_dev/test/pipeline/test-github-members-member-sample.log-expected.json |
Updates expected member relationships. |
packages/github/changelog.yml |
Records the breaking change and teams addition. |
packages/github/_dev/deploy/docker/files/config-teams.yml |
Adds mock GraphQL responses. |
packages/github/_dev/deploy/docker/docker-compose.yml |
Registers the teams mock service. |
packages/github/_dev/build/docs/README.md |
Updates the documentation source template. |
Review effort: Balanced
🚀 Benchmarks reportTo see the full report comment with |
| changes: | ||
| - description: | | ||
| Remove `user.entity.relationships.administers.service.id` and `user.entity.relationships.administers.service.name` from the members data stream. | ||
| Maintainers now set `user.entity.relationships.administers.entity.id` to the GitHub team node id so the Entity Store can join the relationship. | ||
| type: breaking-change | ||
| link: https://github.com/elastic/integrations/pull/22121 | ||
| - description: Add a teams data stream that emits one generic entity document per GitHub organization team. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/22121 |
There was a problem hiding this comment.
Can you add to commit message (short) why both enhancement + breaking-change should coexist together?
| "iam" | ||
| ], | ||
| "kind": "asset", | ||
| "original": "{\"combinedSlug\":\"example-org/frontend-team\",\"databaseId\":1002,\"description\":null,\"id\":\"T_kgDOAADDEE\",\"name\":\"Frontend Team\",\"parentTeam\":{\"id\":\"T_kgDOAABBCC\",\"slug\":\"backend-team\"},\"privacy\":\"SECRET\",\"slug\":\"frontend-team\"}", |
There was a problem hiding this comment.
I know its not a lot of fields and primary usecase is elsewhere (Entity Analytics view), but just for quality can you add a minimal dashboard for this data stream?
| "Authorization": ["Bearer " + state.access_token], | ||
| }, | ||
| "Body": { | ||
| "query": "query OrgTeams($org: String!, $teamCursor: String) { organization(login: $org) { teams(first: 20, after: $teamCursor, orderBy: { field: NAME, direction: ASC }) { pageInfo { hasNextPage endCursor } totalCount edges { node { id databaseId name slug combinedSlug privacy description parentTeam { id slug } } } } } rateLimit { cost remaining resetAt } }", |
There was a problem hiding this comment.
A few Team fields worth adding: url mapped to the ECS entity.reference (currently empty), and createdAt/updatedAt as github.teams.created_at/updated_at.
| "message": "POST /graphql (teams): " + ((body.?errors.orValue(null) != null) ? ((body.errors.size() > 0) ? body.errors[0].?message.orValue("GraphQL error") : "GraphQL error") : "missing organization.teams in response"), | ||
| }, | ||
| }, | ||
| "want_more": false, |
There was a problem hiding this comment.
The error results don't set cursor (here and in the error branch at L87). A failure part-way through therefore leaves cursor.after set, and the next interval resumes mid-roster with a partial snapshot. Add "cursor": {} to both error branches so the next run starts from the beginning.
There was a problem hiding this comment.
🟡 Changes recommended
Team assets lack the required entity.type classification, and the dashboard counts parent slugs rather than stable team IDs, causing renamed parent teams to be overcounted.
2 open issues
2 resolved since last review
Review effort: Balanced
|
✅ All changelog entries have the correct PR link. |
There was a problem hiding this comment.
🔵 Needs a closer look
The ingest fixtures establish event shapes but do not verify that Entity Store materializes the generic team entity and Entity Graph renders its Administers edge in a live cluster.
No open issues 🎉
2 resolved since last review
Review effort: Balanced
|
🟢 No issues across the latest commits dc7265e.
|
💚 Build Succeeded
History
|
|
This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again. |
|
Package github - 3.0.0 containing this change is available at https://epr.elastic.co/package/github/3.0.0/ |

Proposed commit message
The members data stream recorded teams a user maintains under
user.entity.relationships.administers.service.*. The Entity Store kept those
values as raw identifiers and never created a team entity, so the Administers
relationship could not be joined.
Maintainers now set user.entity.relationships.administers.entity.id to the
team GraphQL node id. The previous administers.service.id and
administers.service.name fields are removed. A new github.teams data stream
emits one asset document per organization team, with that same node id in
entity.id, and no user.* fields. github.teams.combined_slug keeps org/team-slug.
The team url is copied to entity.reference. createdAt and updatedAt are stored
as github.teams.created_at and github.teams.updated_at.
Each collection stores a new team
document so entity.last_seen_timestamp advances. The member document id changes
once in 3.0.0 so an unchanged member is indexed again with the new fields.
The Organization Members dashboard terms panel uses
github.members.maintained_team_names. The unique-count panel uses
user.entity.relationships.administers.entity.id. An Organization Teams
dashboard counts distinct teams by entity.id.
Checklist
changelog.ymlfile.How to test this PR locally
From
packages/github, with the Elastic stack up:Related issues
Screenshots