Skip to content

fix(auditd): reduce grok backtracking on split EXECVE argument chunks - #21003

Merged
kcreddy merged 3 commits into
elastic:mainfrom
kcreddy:fix-auditd-execve-grok-backtracking
Sep 2, 2026
Merged

kcreddy merged 3 commits into
elastic:mainfrom
kcreddy:fix-auditd-execve-grok-backtracking

Conversation

@kcreddy

@kcreddy kcreddy commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Proposed commit message

fix(auditd): reduce grok backtracking on split EXECVE argument chunks

Continuation records from auditd's split oversized EXECVE arguments
(aN[0]=, aN[1]=, ...) could not match the grok_audit_original key
pattern because bracket-suffixed keys like a17[1]= were outside
[A-Za-z0-9_]+. On records whose only key has a bracket suffix, the
lazy skip retried a greedy key scan at every offset of a multi-kilobyte
hex value, producing quadratic backtracking that exceeded the grok
watchdog timeout and consumed excessive ingest CPU.

Accept an optional [N] suffix in the AUDIT_KEY_VALUES key class so
split-argument chunk keys match on the first attempt. Keep a bounded
lazy skip ({0,128}?) for records with a short prose preamble before
the first key (TTY, DAEMON_END); removing it breaks field extraction
on those shapes. The longest preamble in pipeline fixtures is 32 chars,
so 128 gives headroom while capping no-key scan cost below the
original unbounded (?:.*?).

Add a pipeline test reproducing the split-EXECVE shape.

Fixes #20966

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Continuation records from auditd's split oversized EXECVE arguments
(aN[0]=, aN[1]=, ...) could not match the grok_audit_original key
pattern because bracket-suffixed keys like a17[1]= were outside
[A-Za-z0-9_]+. On records whose only key has a bracket suffix, the
lazy skip retried a greedy key scan at every offset of a multi-kilobyte
hex value, producing quadratic backtracking that exceeded the grok
watchdog timeout and consumed excessive ingest CPU.
Accept an optional [N] suffix in the AUDIT_KEY_VALUES key class and
bound the lazy skip to 512 characters so records with no parseable key
fail fast. Add a pipeline test reproducing the split-EXECVE shape from
elastic#20966.
On _simulate against a 6.7 KB continuation record, grok time dropped
from ~790 ms to ~7 ms and the chunk key is now captured.

Fixes elastic#20966
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Elastic Docs Style Checker (Vale)

Summary: 1 suggestion found

💡 Suggestions (1): Optional style improvements. Apply when helpful.
File Line Rule Message
packages/auditd/changelog.yml 4 Elastic.Ellipses Use ellipses sparingly. Remove the ellipsis unless it appears in UI text.

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

@kcreddy kcreddy self-assigned this Sep 1, 2026
@kcreddy
kcreddy marked this pull request as ready for review September 1, 2026 18:40
@kcreddy
kcreddy requested a review from a team as a code owner September 1, 2026 18:40
Copilot AI lite review requested due to automatic review settings September 1, 2026 18:40
@kcreddy kcreddy added Integration:auditd Auditd Logs Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Sep 1, 2026
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@kcreddy kcreddy added the bugfix Pull request that fixes a bug issue label Sep 1, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The grok change is narrowly scoped, clearly mitigates the reported backtracking issue, and is supported by added/updated pipeline tests that cover the problematic EXECVE continuation shape.

Pull request overview

This PR addresses a high-CPU ingest failure mode in the auditd integration’s raw ingest pipeline by preventing catastrophic/quadratic grok backtracking on EXECVE continuation records that contain only bracket-suffixed argument keys (e.g., a17[1]=...). It improves both correctness (capturing those keys) and performance (failing fast when no parseable key is found).

Changes:

  • Update the AUDIT_KEY_VALUES grok definition to accept optional [...] numeric suffixes on keys and bound the “lazy skip” to 512 characters.
  • Add a pipeline test fixture for an oversized/split EXECVE argument continuation record and update existing EXECVE truncation expectations accordingly.
  • Bump the auditd package version and add a changelog entry for the fix.
File summaries
File Description
packages/auditd/manifest.yml Bumps package version to 3.24.2.
packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/raw.yml Tightens grok key matching to handle aN[x]= keys and bounds scanning to reduce backtracking.
packages/auditd/data_stream/log/_dev/test/pipeline/test-truncated-execve.log-expected.json Updates expected args extraction now that bracket-suffixed keys can be captured.
packages/auditd/data_stream/log/_dev/test/pipeline/test-oversized-execve.log-expected.json Adds expected output for a new oversized/split EXECVE reproducer.
packages/auditd/data_stream/log/_dev/test/pipeline/test-oversized-execve.log Adds a new test input with a continuation a17[1]=... chunk.
packages/auditd/data_stream/log/_dev/test/pipeline/test-auditd-enriched.log-expected.json Updates expected JSON structure/ordering from regenerated pipeline output.
packages/auditd/changelog.yml Adds a 3.24.2 bugfix entry with the correct PR link.
Review details
  • Files reviewed: 6/7 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

# to 512 characters so that any record with no parseable key after the prefix
# fails fast instead of scanning the whole line; in practice keys start within a
# few characters of the prefix.
AUDIT_KEY_VALUES: '(?:.{0,512}?)(?<auditd.log.kv>[A-Za-z0-9_]+(?:\[[0-9]+\])?=[\s\S]*)'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe the non-capturing group is needed here.

Suggested change
AUDIT_KEY_VALUES: '(?:.{0,512}?)(?<auditd.log.kv>[A-Za-z0-9_]+(?:\[[0-9]+\])?=[\s\S]*)'
AUDIT_KEY_VALUES: '(?<auditd.log.kv>[A-Za-z0-9_]+(?:\[[0-9]+\])?=[\s\S]*)'

When I run the comparison, the pattern without the group costs the same as the bounded-length group.

The question is then how often do we see lines where there are no parseable keys? I've done some experiments and the bounded approach is slower than omitting the capture for long-prefix-without-keys lines.

@kcreddy kcreddy Sep 2, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The skip isn't for the EXECVE case (agreed, the bracket suffix alone covers that) — it's needed for keyed records with a short prose preamble before the first key=. Two lines in test-auditd-raw.log regress without it:

    type=TTY msg=audit(...): tty pid=27930 uid=1000 ...
                             ^^^^ 4-char literal label before pid=
    type=DAEMON_END msg=audit(...): auditd normal halt, sending auid=? pid=? subj=? res=success
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 28 chars of prose before auid=

With the suggested pattern, pattern 1 fails on these, they fall through to %{AUDIT_PREFIX}\s*.*$, and auditd.log.kv is never captured — losing pid, uid, res, etc. Verified with _simulate (2 of 52 lines) and elastic-package test pipeline fails on that fixture.

To address the no-key-line cost, I've tightened the bound from 512 to 128 — the longest preamble in our fixtures is 32 chars, so that's 4× headroom at a quarter of the scan cost. c2b4e74

An alternative is dropping the skip and enumerating the preambles explicitly ((?:tty )?, DAEMON_END prose, ...) — fastest on all paths, but brittle against preamble shapes we haven't catalogued, which is exactly what the skip absorbs.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking into that.

Copilot AI review requested due to automatic review settings September 2, 2026 05:22
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

✅ All changelog entries have the correct PR link.

@kcreddy
kcreddy requested a review from efd6 September 2, 2026 05:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The PR description states a 512-character bound but the pipeline currently uses 128, so the implementation and documented intent should be aligned to avoid confusion and potential edge-case parsing failures.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 6/7 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment on lines +55 to +59
# to 128 characters: the longest no-key preamble in pipeline fixtures is 32 chars
# (DAEMON_END, unmatched AVC preamble), so 128 covers real shapes with headroom
# while capping O(n²) scan cost on malformed no-key lines below an unbounded
# (?:.*?).
AUDIT_KEY_VALUES: '(?:.{0,128}?)(?<auditd.log.kv>[A-Za-z0-9_]+(?:\[[0-9]+\])?=[\s\S]*)'
@vera-review-bot

Copy link
Copy Markdown
Contributor

🟢 No issues across the latest commits c2b4e74.

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.3.0 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

💚 Build Succeeded

History

cc @kcreddy

@kcreddy
kcreddy merged commit b3ad795 into elastic:main Sep 2, 2026
11 checks passed
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package auditd - 3.24.2 containing this change is available at https://epr.elastic.co/package/auditd/3.24.2/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Pull request that fixes a bug issue Integration:auditd Auditd Logs Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

⌂ 换网址