Skip to content

fix: close HTTP response bodies to prevent OOM with many checks - #1058

Merged
aelsabbahy merged 1 commit into
goss-org:masterfrom
dukelion:fix/http-oom-response-body-cleanup
Jul 5, 2026
Merged

aelsabbahy merged 1 commit into
goss-org:masterfrom
dukelion:fix/http-oom-response-body-cleanup

Conversation

@dukelion

@dukelion dukelion commented May 7, 2026 •

Copy link
Copy Markdown
Contributor

When running 100+ HTTP checks concurrently, response bodies were not always closed after validation. This caused unclosed TCP connections to accumulate, holding TLS state and transport buffers until the client timeout expired, leading to OOM kills under memory limits.

Checklist
  • make test-all (UNIX) passes. CI will also test this
  • unit and/or integration tests are included (if applicable)
  • documentation is changed or added (if applicable)

Description of change

Fixes OOM when goss is configured with many HTTP checks and limited RAM.

  • Add Close() to the system.HTTP interface and DefHTTP
  • Defer sysHTTP.Close() from HTTP resource validation for guaranteed cleanup
  • Always close resp.Body when Close() is called, including after a body matcher reads it
  • Add a unit test covering close-after-body-read behavior

@dukelion
dukelion requested a review from aelsabbahy as a code owner May 7, 2026 10:00
@kgaughan

kgaughan commented Jun 8, 2026 •

Copy link
Copy Markdown
Collaborator

Hi! I fixed the integration test pipelines recently. Could you rebase or merge from master? I checked you integration test failures, and most if not all of them would be fixed by that.

When running 100+ HTTP checks concurrently, response bodies were never
closed if the check only validated status and headers (no body matcher).
This caused unclosed TCP connections to accumulate, holding TLS state
and transport buffers until the 10s client timeout expired, leading to
OOM kills under memory limits.

Changes:
- Add Close() to system.HTTP interface and DefHTTP
- Track whether Body() consumed the response body
- Close() closes resp.Body if it wasn't consumed (i.e. status/headers-only checks)
- resource/http.go Validate() defers sysHTTP.Close() for guaranteed cleanup

Fixes OOM when goss is configured with many HTTP checks and limited RAM.
@dukelion
dukelion force-pushed the fix/http-oom-response-body-cleanup branch from 54592bc to b13b130 Compare June 9, 2026 09:24
@dukelion

dukelion commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

Rebased this branch onto current goss-org/master and force-pushed with lease.

I also tightened the cleanup path so DefHTTP.Close() closes resp.Body even after a body matcher reads it, and added a focused unit test for that lifecycle.

Local validation run on Windows:

  • go test ./system -run TestDefHTTPCloseClosesBodyAfterBodyRead
  • go test ./resource
  • git diff --check HEAD~1..HEAD

@leemeo3

leemeo3 commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

I took a look at the current failing Ubuntu check for this PR.

Evidence:

  • Failing job: Integration tests (ubuntu-latest) in https://github.com/goss-org/goss/actions/runs/27196730484/job/85238831313
  • Failing command path: make test-int-all -> cd integration-tests/ && ./test.sh bullseye 386
  • In this PR run, integration-tests/test.sh could not find Dockerfile_bullseye.md5, rebuilt ghcr.io/goss-org/goss_integration_bullseye:latest, and Debian bullseye currently installs apache2 2.4.67-1~deb11u3.
  • The validation still expects 2.4.67-1~deb11u2, so bullseye-32 fails with Count: 127, Failed: 1, Skipped: 5.
  • This PR's own diff only touches resource/http.go, system/http.go, and system/http_test.go; I locally checked the focused test from the PR with go test ./system -run TestDefHTTPCloseClosesBodyAfterBodyRead, and it passes.
  • Current master has a passing Integration tests run at https://github.com/goss-org/goss/actions/runs/28747084644, and its integration-tests/test.sh now unconditionally pulls the published integration image instead of rebuilding it locally.

Given that, this looks more like the branch being behind current master integration-test infrastructure than a failure caused by the HTTP response-body cleanup itself. A rebase onto current master and rerun of CI looks like the least invasive next step.

I did not open a separate PR because changing the integration test golden package versions or image behavior would be broader than this PR's HTTP cleanup scope and may overlap with the ongoing release/pipeline work.

AI assistance disclosure: I used Codex to help inspect logs and compare branches, then reviewed the evidence and local command results before posting.

@kgaughan

kgaughan commented Jul 5, 2026

Copy link
Copy Markdown
Collaborator

The problem is the lack is the .md5 file. That's used to determine if an image should be built from scratch. It's an easy fix, but annoying that I missed it when fixing the integration tests.

@kgaughan

kgaughan commented Jul 5, 2026

Copy link
Copy Markdown
Collaborator

I know what the issue is. @dukelion or @aelsabbahy, could one of you merge from master? The issue is that this PR is still using an older version of integration-tests/test.sh.

@aelsabbahy

Copy link
Copy Markdown
Member

Merging based on discussion here: #1049 (comment)

@aelsabbahy
aelsabbahy merged commit 906c395 into goss-org:master Jul 5, 2026
5 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

⌂ 换网址