Repository navigation
[Threat Intel] Remove redundant ECS field definitions from transform destination indices - section C - #21624
Conversation
Kibana now composes the `ecs@mappings` component template into transform destination index templates (>= 8.19.0 / 9.1.0, elastic/kibana#177608), so ECS fields no longer need to be declared explicitly in transform field files, and the package-level `strings_as_keyword` catch-all dynamic template is no longer required. Across the Section C threat intelligence integrations this removes the redundant ECS definitions and the catch-all, keeping only: - non-ECS fields (agent.yml / beats.yml, e.g. cloud.image.id, host.containerized, input.type), - base routing/timestamp fields (base-fields.yml), - ECS overrides with a constant value or a custom definition (ecs-overridden.yml). The Kibana constraints are already at `^8.19.x || ^9.1.x` or later, so no constraint bump is needed; only the transform destination index version is bumped so corrected mappings apply on upgrade.
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
✅ Elastic Docs Style Checker (Vale)No issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
|
🟢 No issues across the latest commits 2cb87cb.
🤖 AI-Generated Review | Vera Review Bot - v0.4.2 | 📚 Knowledge base: integration-skills
|
🚀 Benchmarks reportTo see the full report comment with |
kcreddy
left a comment
There was a problem hiding this comment.
LGTM, please fix the conflicts before merging
|
✅ All changelog entries have the correct PR link. |
💚 Build Succeeded
History
|
|
This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again. |
Backport branchesImportant Only active backport branches for packages changed by this PR are shown. Tick the branches you want to backport to. A backport PR will be opened automatically for each branch you check. Backport a change when it fixes behavior a branch already has; leave new behavior on ti_abusech
Tip If a branch above is no longer required, set |
|
Package ti_abusech - 5.2.0 containing this change is available at https://epr.elastic.co/package/ti_abusech/5.2.0/ |
|
Package ti_custom - 1.10.0 containing this change is available at https://epr.elastic.co/package/ti_custom/1.10.0/ |
|
Package ti_otx - 1.35.0 containing this change is available at https://epr.elastic.co/package/ti_otx/1.35.0/ |
|
Package ti_threatconnect - 2.6.0 containing this change is available at https://epr.elastic.co/package/ti_threatconnect/2.6.0/ |
…22123) ti_*: statically map event.ingested in transform destination indices The section A/B/C cleanup (#21585, #21596, #21624) removed the event.ingested definition from every TI transform and left it to ecs@mappings. That template only maps the field dynamically, when a document containing it is indexed, so a destination index whose source data stream is empty (for example, a disabled data stream) has no event.ingested mapping. Detection rules that use event.ingested as the timestamp override then report those indices as missing the field. This is the same problem #14082 fixed for ti_anomali. Add a fields/ecs.yml declaring event.ingested (external: ecs) to all 64 TI transforms, including the 7 that sync on @timestamp, because the rule check depends on the rule's index pattern, not the transform's sync field. Each file carries a comment saying why the field must stay statically mapped, so a later cleanup does not remove it again. Bump each destination index suffix and fleet_transform_version so the mapping applies on upgrade.
Proposed commit message
Checklist
changelog.ymlfile.Author's Checklist
ecs@mappingsfor special ECS types (e.g.url.full/url.original-> wildcard, dates, ip).How to test this PR locally
elastic-packagelocally.elastic-package(8.19.0 or 9.1.0).ti_abusech,ti_threatconnect):integrations/packages/<package>.elastic-package buildandelastic-package lint.elastic-package install.event.module/event.dataset(frombase-fields.yml) are present asconstant_keywordwith the correct values,threat.feed.*overrides are present, non-ECS agent/beats fields keep their types, package fields are present, and theecs@mappingsdynamic templates are composed in (all_strings_to_keywords,ecs_ip,ecs_geo_point,ecs_message_match_only_text,ecs_timestamp).Related issues