Skip to content

[Threat Intel] Remove redundant ECS field definitions from transform destination indices - section C - #21624

Merged
brijesh-elastic merged 3 commits into
elastic:mainfrom
brijesh-elastic:ti-remove_redudent_ECS-C
Oct 12, 2026
Merged

brijesh-elastic merged 3 commits into
elastic:mainfrom
brijesh-elastic:ti-remove_redudent_ECS-C

Conversation

@brijesh-elastic

Copy link
Copy Markdown
Contributor

Proposed commit message

ti: remove redundant ECS field definitions from transforms

Kibana composes the `ecs@mappings` component template into transform
destination index templates on supported stacks (>= 8.19.0 / 9.1.0), and
elastic/kibana#177608 fixed transform field loading so all fields files are
merged together like data streams instead of a shallow, file-by-file merge.
As a result, ECS fields no longer need to be redeclared in each transform's
field files, and the package-level `strings_as_keyword` catch-all dynamic
template is no longer required (it shadowed ECS special-case mappings such
as `message` -> match_only_text, `event.original` -> non-indexed, and
`*url.full` / `*url.original` -> wildcard).

Across the Section C threat intelligence integrations this removes the
redundant ECS definitions and the catch-all, and bumps the transform
destination index version so corrected mappings apply on upgrade. These
packages already pin Kibana to `^8.19.x || ^9.1.x` (or later), so no
constraint change is required.

What is kept and why:
- Non-ECS fields stay declared (ecs@mappings does not define them):
  agent.yml (cloud.image.id, host.containerized, host.os.build,
  host.os.codename), beats.yml (input.type, log.offset, log.flags,
  log.file.device_id/fingerprint/inode) and custom package fields
  (e.g. ti_otx threat.indicator.file.hash.pehash).
- Base routing/timestamp fields remain in base-fields.yml (data_stream.*,
  event.module, event.dataset, @timestamp). event.module/event.dataset were
  moved here from ecs.yml (flagged "to be moved into base-fields.yml after
  kibana.version >= 8.14") for ti_abusech and ti_threatconnect.; ti_custom already
  had them in base-fields.yml.
- ECS overrides (constant value) moved to ecs-overridden.yml:
  - ti_abusech: threat.feed.name, threat.feed.dashboard_id (per transform).
  - ti_threatconnect: threat.feed.name.
  - ti_otx: threat.feed.name, threat.feed.dashboard_id, threat.indicator.file.hash.pehash.

Packages updated:
- ti_abusech 5.2.0
- ti_custom 1.10.0
- ti_otx 1.35.0
- ti_threatconnect 2.6.0

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • Confirm the destination index version bump on each transform so corrected mappings are applied on upgrade.
  • Confirm non-ECS fields (agent.yml/beats.yml) and ECS overrides (ecs-overridden.yml) are retained per package.
  • Confirm ti_custom relies on ecs@mappings for special ECS types (e.g. url.full/url.original -> wildcard, dates, ip).

How to test this PR locally

  • Clone the integrations repo.
  • Install elastic-package locally.
  • Start the Elastic stack using elastic-package (8.19.0 or 9.1.0).
  • For a changed package (e.g. ti_abusech, ti_threatconnect):
    • Move to integrations/packages/<package>.
    • Run elastic-package build and elastic-package lint.
    • Run elastic-package install.
    • Query the transform destination index mapping and confirm fields from all fields files coexist and are typed correctly:

GET //_mapping

  • Verify event.module/event.dataset (from base-fields.yml) are present as constant_keyword with the correct values, threat.feed.* overrides are present, non-ECS agent/beats fields keep their types, package fields are present, and the ecs@mappings dynamic templates are composed in (all_strings_to_keywords, ecs_ip, ecs_geo_point, ecs_message_match_only_text, ecs_timestamp).

Related issues

Kibana now composes the `ecs@mappings` component template into transform
destination index templates (>= 8.19.0 / 9.1.0, elastic/kibana#177608),
so ECS fields no longer need to be declared explicitly in transform field
files, and the package-level `strings_as_keyword` catch-all dynamic
template is no longer required.

Across the Section C threat intelligence integrations this removes the
redundant ECS definitions and the catch-all, keeping only:
- non-ECS fields (agent.yml / beats.yml, e.g. cloud.image.id,
  host.containerized, input.type),
- base routing/timestamp fields (base-fields.yml),
- ECS overrides with a constant value or a custom definition
  (ecs-overridden.yml).

The Kibana constraints are already at `^8.19.x || ^9.1.x` or later, so no
constraint bump is needed; only the transform destination index version is
bumped so corrected mappings apply on upgrade.
@brijesh-elastic brijesh-elastic self-assigned this Sep 24, 2026
@brijesh-elastic
brijesh-elastic requested review from a team as code owners September 24, 2026 07:10
@brijesh-elastic brijesh-elastic added enhancement New feature or request Integration:ti_abusech abuse.ch Integration:ti_otx AlienVault OTX Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Integration:ti_threatconnect ThreatConnect Integration:ti_custom Custom Threat Intelligence Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] labels Sep 24, 2026
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@github-actions

Copy link
Copy Markdown
Contributor

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!


The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

@vera-review-bot

Copy link
Copy Markdown
Contributor

🟢 No issues across the latest commits 2cb87cb.

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.4.2 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@kcreddy kcreddy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please fix the conflicts before merging

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

✅ All changelog entries have the correct PR link.

@brijesh-elastic
brijesh-elastic enabled auto-merge (squash) October 12, 2026 06:36
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

💚 Build Succeeded

History

cc @brijesh-elastic

@mergify

mergify Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again.

@brijesh-elastic
brijesh-elastic merged commit 295bf9e into elastic:main Oct 12, 2026
14 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport branches

Important

Only active backport branches for packages changed by this PR are shown.

Tick the branches you want to backport to. A backport PR will be opened automatically for each branch you check.

Backport a change when it fixes behavior a branch already has; leave new behavior on main. See when and why to backport if you are unsure.

ti_abusech

  • backport-ti_abusech-2.6

Tip

If a branch above is no longer required, set archived: true in its entry in .backports.yml to stop it appearing here.
If the branch has a known end-of-life date, prefer maintained_until: "YYYY-MM-DD" — it will be excluded automatically once that date passes.

cc @brijesh-elastic

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package ti_abusech - 5.2.0 containing this change is available at https://epr.elastic.co/package/ti_abusech/5.2.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package ti_custom - 1.10.0 containing this change is available at https://epr.elastic.co/package/ti_custom/1.10.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package ti_otx - 1.35.0 containing this change is available at https://epr.elastic.co/package/ti_otx/1.35.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package ti_threatconnect - 2.6.0 containing this change is available at https://epr.elastic.co/package/ti_threatconnect/2.6.0/

kcreddy added a commit that referenced this pull request Oct 27, 2026
…22123)

ti_*: statically map event.ingested in transform destination indices

The section A/B/C cleanup (#21585, #21596, #21624) removed the
event.ingested definition from every TI transform and left it to
ecs@mappings. That template only maps the field dynamically, when a
document containing it is indexed, so a destination index whose
source data stream is empty (for example, a disabled data stream)
has no event.ingested mapping. Detection rules that use
event.ingested as the timestamp override then report those indices
as missing the field. This is the same problem #14082 fixed for
ti_anomali.

Add a fields/ecs.yml declaring event.ingested (external: ecs) to all
64 TI transforms, including the 7 that sync on @timestamp, because
the rule check depends on the rule's index pattern, not the
transform's sync field. Each file carries a comment saying why the
field must stay statically mapped, so a later cleanup does not
remove it again. Bump each destination index suffix and
fleet_transform_version so the mapping applies on upgrade.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Integration:ti_abusech abuse.ch Integration:ti_custom Custom Threat Intelligence Integration:ti_otx AlienVault OTX Integration:ti_threatconnect ThreatConnect Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

⌂ 换网址