Repository navigation
fix(auditd): reduce grok backtracking on split EXECVE argument chunks - #21003
Conversation
Continuation records from auditd's split oversized EXECVE arguments (aN[0]=, aN[1]=, ...) could not match the grok_audit_original key pattern because bracket-suffixed keys like a17[1]= were outside [A-Za-z0-9_]+. On records whose only key has a bracket suffix, the lazy skip retried a greedy key scan at every offset of a multi-kilobyte hex value, producing quadratic backtracking that exceeded the grok watchdog timeout and consumed excessive ingest CPU. Accept an optional [N] suffix in the AUDIT_KEY_VALUES key class and bound the lazy skip to 512 characters so records with no parseable key fail fast. Add a pipeline test reproducing the split-EXECVE shape from elastic#20966. On _simulate against a 6.7 KB continuation record, grok time dropped from ~790 ms to ~7 ms and the chunk key is now captured. Fixes elastic#20966
Elastic Docs Style Checker (Vale)Summary: 1 suggestion found 💡 Suggestions (1): Optional style improvements. Apply when helpful.
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
There was a problem hiding this comment.
🟢 Approval recommended
The grok change is narrowly scoped, clearly mitigates the reported backtracking issue, and is supported by added/updated pipeline tests that cover the problematic EXECVE continuation shape.
Pull request overview
This PR addresses a high-CPU ingest failure mode in the auditd integration’s raw ingest pipeline by preventing catastrophic/quadratic grok backtracking on EXECVE continuation records that contain only bracket-suffixed argument keys (e.g., a17[1]=...). It improves both correctness (capturing those keys) and performance (failing fast when no parseable key is found).
Changes:
- Update the
AUDIT_KEY_VALUESgrok definition to accept optional[...]numeric suffixes on keys and bound the “lazy skip” to 512 characters. - Add a pipeline test fixture for an oversized/split EXECVE argument continuation record and update existing EXECVE truncation expectations accordingly.
- Bump the auditd package version and add a changelog entry for the fix.
File summaries
| File | Description |
|---|---|
| packages/auditd/manifest.yml | Bumps package version to 3.24.2. |
| packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/raw.yml | Tightens grok key matching to handle aN[x]= keys and bounds scanning to reduce backtracking. |
| packages/auditd/data_stream/log/_dev/test/pipeline/test-truncated-execve.log-expected.json | Updates expected args extraction now that bracket-suffixed keys can be captured. |
| packages/auditd/data_stream/log/_dev/test/pipeline/test-oversized-execve.log-expected.json | Adds expected output for a new oversized/split EXECVE reproducer. |
| packages/auditd/data_stream/log/_dev/test/pipeline/test-oversized-execve.log | Adds a new test input with a continuation a17[1]=... chunk. |
| packages/auditd/data_stream/log/_dev/test/pipeline/test-auditd-enriched.log-expected.json | Updates expected JSON structure/ordering from regenerated pipeline output. |
| packages/auditd/changelog.yml | Adds a 3.24.2 bugfix entry with the correct PR link. |
Review details
- Files reviewed: 6/7 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| # to 512 characters so that any record with no parseable key after the prefix | ||
| # fails fast instead of scanning the whole line; in practice keys start within a | ||
| # few characters of the prefix. | ||
| AUDIT_KEY_VALUES: '(?:.{0,512}?)(?<auditd.log.kv>[A-Za-z0-9_]+(?:\[[0-9]+\])?=[\s\S]*)' |
There was a problem hiding this comment.
I don't believe the non-capturing group is needed here.
| AUDIT_KEY_VALUES: '(?:.{0,512}?)(?<auditd.log.kv>[A-Za-z0-9_]+(?:\[[0-9]+\])?=[\s\S]*)' | |
| AUDIT_KEY_VALUES: '(?<auditd.log.kv>[A-Za-z0-9_]+(?:\[[0-9]+\])?=[\s\S]*)' |
When I run the comparison, the pattern without the group costs the same as the bounded-length group.
The question is then how often do we see lines where there are no parseable keys? I've done some experiments and the bounded approach is slower than omitting the capture for long-prefix-without-keys lines.
There was a problem hiding this comment.
The skip isn't for the EXECVE case (agreed, the bracket suffix alone covers that) — it's needed for keyed records with a short prose preamble before the first key=. Two lines in test-auditd-raw.log regress without it:
type=TTY msg=audit(...): tty pid=27930 uid=1000 ...
^^^^ 4-char literal label before pid=
type=DAEMON_END msg=audit(...): auditd normal halt, sending auid=? pid=? subj=? res=success
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 28 chars of prose before auid=
With the suggested pattern, pattern 1 fails on these, they fall through to %{AUDIT_PREFIX}\s*.*$, and auditd.log.kv is never captured — losing pid, uid, res, etc. Verified with _simulate (2 of 52 lines) and elastic-package test pipeline fails on that fixture.
To address the no-key-line cost, I've tightened the bound from 512 to 128 — the longest preamble in our fixtures is 32 chars, so that's 4× headroom at a quarter of the scan cost. c2b4e74
An alternative is dropping the skip and enumerating the preambles explicitly ((?:tty )?, DAEMON_END prose, ...) — fastest on all paths, but brittle against preamble shapes we haven't catalogued, which is exactly what the skip absorbs.
|
✅ All changelog entries have the correct PR link. |
There was a problem hiding this comment.
🟡 Changes recommended
The PR description states a 512-character bound but the pipeline currently uses 128, so the implementation and documented intent should be aligned to avoid confusion and potential edge-case parsing failures.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 6/7 changed files
- Comments generated: 1
- Review effort level: Lite
| # to 128 characters: the longest no-key preamble in pipeline fixtures is 32 chars | ||
| # (DAEMON_END, unmatched AVC preamble), so 128 covers real shapes with headroom | ||
| # while capping O(n²) scan cost on malformed no-key lines below an unbounded | ||
| # (?:.*?). | ||
| AUDIT_KEY_VALUES: '(?:.{0,128}?)(?<auditd.log.kv>[A-Za-z0-9_]+(?:\[[0-9]+\])?=[\s\S]*)' |
|
🟢 No issues across the latest commits c2b4e74.
🤖 AI-Generated Review | Vera Review Bot - v0.3.0 | 📚 Knowledge base: integration-skills
|
🚀 Benchmarks reportTo see the full report comment with |
💚 Build Succeeded
History
cc @kcreddy |
|
Package auditd - 3.24.2 containing this change is available at https://epr.elastic.co/package/auditd/3.24.2/ |
Proposed commit message
Checklist
changelog.ymlfile.