Skip to content

Open SFTP and SSH store connections on demand - #432

Merged
folbricht merged 2 commits into
masterfrom
lazy-store-pools
Sep 20, 2026
Merged

folbricht merged 2 commits into
masterfrom
lazy-store-pools

Conversation

@folbricht

@folbricht folbricht commented Sep 20, 2026 •

Copy link
Copy Markdown
Owner

The SFTP and SSH stores opened all N connections when the store was created, whether they were needed or not. With a high -n that meant starting that many SSH processes up front. This is also a prerequisite for adaptive concurrency (#430), where the number of requests in flight changes during a run and the connections should follow it.

Both stores now share a small pool (sessionpool.go) that:

  • opens the first connection when the store is created, so a wrong host or path still fails at startup
  • opens further connections only when all existing ones are busy, up to N, and blocks callers once N are in use
  • opens connections one at a time, since SSH may fail when several processes use the SSH agent concurrently
  • waits for all connections to be returned on Close, and reports every close error instead of only the last one

A failure to open a later connection is now returned by the request that needed it rather than when the store is created. A pool size below 1 is treated as 1; previously a store created with N: 0 blocked forever on its first request.

The S3 store kept minio's default of 16 idle connections per host, so with a concurrency above 16 the extra connections were closed and reopened, with a new TLS handshake, for every request. It now keeps one per concurrent request, as the HTTP and OCI stores already do. Raising the per-host limit alone isn't enough, since minio also caps idle connections at 256 in total, so that cap is raised too. The OCI store had the same problem above 100, the total cap of the Go default transport it clones, and gets the same fix. GCS already keeps 100 per host and is unchanged.

The SFTP and SSH stores opened all N connections when the store was
created, whether they were needed or not. They now share a pool that
opens the first connection up front, to confirm the store can be
reached, and further ones only when all existing connections are busy,
up to N. Connections are still opened one at a time.

The S3 store kept minio's default of 16 idle connections per host, so
with a concurrency above 16 the extra connections were closed and
reopened for every request. It now keeps one per concurrent request.
Raising the per-host idle limit isn't enough on its own. minio's
transport caps idle connections at 256 in total and Go's default
transport, which the OCI store clones, at 100. Above that, connections
were still closed and reopened for every request.
@folbricht
folbricht merged commit f22b340 into master Sep 20, 2026
9 checks passed
@folbricht
folbricht deleted the lazy-store-pools branch September 20, 2026 16:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

⌂ 换网址