Skip to content

feat(storage): log transient errors retried on the sync paths - #16467

Merged
v-pratap merged 1 commit into
googleapis:mainfrom
v-pratap:retry-logging-sync
Sep 22, 2026
Merged

v-pratap merged 1 commit into
googleapis:mainfrom
v-pratap:retry-logging-sync

Conversation

@v-pratap

@v-pratap v-pratap commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem

When GCS returns a transient error (429, 503, 500, DEADLINE_EXCEEDED) the
client backs off and retries with no output at all. From the application's
point of view a read just takes a long time.

There is nothing to grep for, so an operator cannot tell a slow network from a
server that is shedding load. The only way to see the retries today is to enable
the rawclient tracing component and pay for a full dump of every request and
response — which you can only do after you already suspect retries, i.e. not
during the first occurrence of an incident.

This came out of a real incident where reads on a large fleet appeared to stall
and the client produced nothing to correlate against.

What this does

One WARNING record per retried attempt, nothing on success:

[gcs-retry] <where>: transient error on attempt <n>, resource=<bucket/object>, status=<status>

This PR covers the synchronous paths, which serve both the JSON/REST and the
gRPC transports — GCS gRPC sync also goes through RestRetryLoop, because
GenericStub takes a RestContext& regardless of transport.

Call site where Fires when
ConnectionImpl::ReadObject (retry lambda) ReadObject/open an attempt failed to open the stream
RetryObjectReadSource::Read ReadObject/resume the stream broke mid-download and is resumed from the last received byte
ConnectionImpl::UploadChunk UploadChunk an attempt to upload a chunk of a resumable upload failed

New helper google/cloud/storage/internal/retry_logging.{h,cc}
(LogTransientRetry, LoggingRetryPolicy, RetryLogResource,
RetryLogUploadResource) so the call sites stay one line each.

Also makes LoggingStub::ReadObject report the status on failure. It cannot use
LogWrapper because unique_ptr<ObjectReadSource> has no operator<<, so it
previously logged nothing about why the call failed.

Design notes

Why WARNING, and why not behind a tracing component

LogSink::minimum_severity_ defaults to TRACE, but the default
StdClogBackend drops everything below FATAL
(log.cc#L228-L231).
So this is invisible in a default OSS build and becomes visible to anyone
who installs their own LogBackend. A diagnostic that has to be turned on first
is not useful during the first occurrence of an incident, which is exactly when
it is needed.

No changes to google/cloud/internal/** or log.cc; everything is inside
google/cloud/storage/**.

The status is flattened to one line

GCS error bodies end with a newline. Streaming the Status straight into
GCP_LOG carried that newline through, pushing the backend's (file:line)
suffix onto a second line — so grep '\[gcs-retry\]' returned a truncated
record, defeating the point of the greppable prefix.

It is also a small log-injection hazard, since the message is
service-supplied. OneLine() flattens \n, \r, \t.
Pinned by RetryLogging.StatusIsFlattenedToASingleLine.

The upload session URL is redacted

UploadChunkRequest only exposes upload_session_url(), and the upload_id in
it is a bearer capability — anyone holding it can append to or finalize that
upload with no other credential, for the lifetime of the session (~7 days).

The request's own operator<< does print it, but that is behind the opt-in
rawclient component, whereas this record is always on and lands in log
aggregators that are usually readable by more people than the bucket itself.

RetryLogUploadResource() keeps the bucket and an 8-character prefix of the id
— enough to tell concurrent uploads apart, not enough to hijack one.

Both transports are handled, because UploadChunkRequest carries whichever form
the transport produced: REST passes a URL with an upload_id= query parameter,
gRPC may pass projects/<project>/buckets/<bucket>/<token>, where the token may
itself contain /. Anything matching neither shape is redacted whole, on
the grounds that gRPC also issues wholly opaque tokens (see the testbench
section) and treating an unfamiliar value as safe would leak the entire
capability the first time the service changed format.

Every record comes from the retry policy in force

ReadObject/open has no natural place to count attempts: the work happens inside
a lambda that RestRetryLoop() calls. Rather than classifying the status
separately, it wraps the caller's policy in LoggingRetryPolicy, a
google::cloud::RetryPolicy decorator whose OnFailure() asks the wrapped
policy first and logs only when the answer is "retry".

A record is therefore emitted only when another attempt really does follow. A
permanent error, the final attempt of an exhausted policy, and a non-idempotent
request are all silent — each is already reported to the caller through the
returned Status. There is also no second copy of "which errors are retryable"
to drift away from the policy, so a caller who overrides RetryPolicyOption
automatically gets matching logs.

ReadObject/resume and UploadChunk already sat after their policy's
OnFailure(), so all three sites are now exact.

The backoff delay is deliberately not logged: reading it requires
backoff_policy.OnCompletion(), which advances the policy.

Testbench verification

End-to-end against
storage-testbench v0.61.0
using its fault-injection API (POST /retry_test, id passed as
x-retry-test-id on REST and as call metadata on gRPC), with
GOOGLE_CLOUD_CPP_ENABLE_CLOG=yes. Re-run in full after switching to the
LoggingRetryPolicy decorator. Each scenario asserts an exact record count.

# Scenario Fault injected Expected Result
1 REST read, policy exhausted objects.get: 503 ×3, 2 retries 2 PASS
2 REST read, zero-retry policy objects.get: 503, 0 retries 0 PASS
3 REST read, retries succeed objects.get: 503 ×2, 2 retries 2 PASS
4 REST read, permanent error objects.get: 400 0 PASS
5 Control — healthy read none 0 PASS
6 REST resumable upload objects.insert: return-503-after-256K 1 PASS
7 gRPC resumable upload objects.insert: return-503-after-256K 1 PASS

Scenarios 1 and 2 are the ones the decorator changed. Previously the give-up
attempt was reported as though it were retried, and a policy granting no
retries still produced a record.

===== 1. REST read, policy exhausted (3x503, 2 retries) -- give-up attempt must be silent
  [WARNING] [gcs-retry] ReadObject/open: transient error on attempt 1, resource=retry-logging-demo-bucket/small.txt, status=UNAVAILABLE: {"error":{"code":"503","message":{"error":{"message":"Retry Test: Caused a 503"}}}} (google/cloud/storage/internal/retry_logging.cc:144)
  [WARNING] [gcs-retry] ReadObject/open: transient error on attempt 2, resource=retry-logging-demo-bucket/small.txt, status=UNAVAILABLE: {"error":{"code":"503","message":{"error":{"message":"Retry Test: Caused a 503"}}}} (google/cloud/storage/internal/retry_logging.cc:144)
  --> 2 record(s), expected 2  [PASS]

===== 2. REST read, zero-retry policy (1x503, 0 retries) -- must be silent
  --> 0 record(s), expected 0  [PASS]

===== 4. REST read, permanent error -- must be silent
  --> 0 record(s), expected 0  [PASS]

===== 5. CONTROL: healthy REST read -- must be silent
  --> 0 record(s), expected 0  [PASS]

===== 6. REST resumable upload -- REST upload_id redacted
  [WARNING] [gcs-retry] UploadChunk: transient error on attempt 1, resource=http://localhost:9000/upload/storage/v1/b/retry-logging-demo-bucket/o?uploadType=resumable&upload_id=a40d2c89...[redacted], status=UNAVAILABLE: {"error":{"code":503,"message":"Fault injected during a resumable upload"}} (google/cloud/storage/internal/retry_logging.cc:144)
  --> 1 record(s), expected 1  [PASS]

===== 7. gRPC resumable upload -- opaque session token redacted
  [WARNING] [gcs-retry] UploadChunk: transient error on attempt 1, resource=7d7ab233...[redacted], status=UNAVAILABLE: {"error": {"errors": [{"domain": "global", "message": "Retry Test: Caused a StatusCode.UNAVAILABLE..."}]}} (google/cloud/storage/internal/retry_logging.cc:144)
  --> 1 record(s), expected 1  [PASS]

The trailing (retry_logging.cc:144) is added by the library's own log backend
(operator<<(std::ostream&, LogRecord const&) in log.cc), not by this change.

Important

Scenario 7 is why this section exists. The gRPC upload session id is a bare
opaque 64-character hex string, not the projects/_/buckets/<bucket>/<token>
resource name. The earlier code returned unrecognised shapes unchanged and so
logged the whole token — a bearer credential — in clear. Redaction is now
fail-closed: anything not structurally recognised is redacted whole, keeping
only a short prefix for correlation. No unit test caught this.

Tests

bazel test //google/cloud/storage:all → 175 / 175 PASSED on this branch
alone.

New coverage in retry_logging_test.cc, connection_impl_object_test.cc,
retry_object_read_source_test.cc, logging_stub_test.cc — including negative
tests asserting nothing is logged on success, on a permanent error, on the
final attempt of an exhausted policy, or on a non-idempotent request.

UploadChunkLogsTransientFailures now uses a realistic resumable session URL
and asserts the bucket survives while the upload_id does not.

Follow-ups

  • The async and bidi paths are handled in a stacked PR on top of this one.
  • Write coverage here is deliberately partial: only UploadChunk. A
    RetryLoggingStub GenericStub decorator installed in
    generic_stub_factory.cc would cover every unary RPC from a single place;
    worth doing separately rather than growing this PR.
  • Log volume was raised in review and settled: one line per retried attempt is
    kept. ExponentialBackoffPolicy already spaces attempts exponentially in
    wall-clock time, and the decorator bounds the count at exactly the number of
    attempts the policy granted.

Internal ref: b/547926601

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new retry logging mechanism (retry_logging.h and retry_logging.cc) to provide diagnostics for silently retried transient errors in GCS operations, integrating it into ReadObject, UploadChunk, and RetryObjectReadSource. Feedback on the changes highlights style guide violations, specifically recommending the use of std::string_view instead of absl::string_view and avoiding auto for primitive types.

@v-pratap
v-pratap marked this pull request as ready for review September 21, 2026 05:07
@v-pratap
v-pratap requested review from a team as code owners September 21, 2026 05:07
@codecov

codecov Bot commented Sep 21, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.48718% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.35%. Comparing base (4365822) to head (98cadb6).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...ud/storage/internal/connection_impl_object_test.cc 98.85% 1 Missing ⚠️
...oogle/cloud/storage/internal/retry_logging_test.cc 99.27% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #16467      +/-   ##
==========================================
+ Coverage   92.34%   92.35%   +0.01%     
==========================================
  Files        2249     2252       +3     
  Lines      214649   215038     +389     
==========================================
+ Hits       198212   198601     +389     
  Misses      16437    16437              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@kalragauri kalragauri left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding your open question on log volume, logging every retried attempt (once ReadObject/open is gated on retry_policy.OnFailure() returning true) is the right choice IMHO rather than sampling powers of two. IIUC, ExponentialBackoffPolicy (the default retry policy) already spaces consecutive attempts exponentially in wall-clock time.

@v-pratap

Copy link
Copy Markdown
Contributor Author

Thanks — agreed, and I have dropped the throttling idea. Your reasoning holds up when I check it against the code: ExponentialBackoffPolicy::OnCompletion() scales the delay by scaling_ each time, and the storage defaults are 1s initial / 5min maximum / 2x. So the records are already self-limiting in wall-clock terms; the interval between them grows until it hits the 5 minute ceiling. Sampling on top of that would mostly hide the early attempts, which are the ones an operator wants when deciding whether a spike is service-side.

The decorator you asked for in the ReadObject/open thread also bounds the total directly: the count is now exactly the number of attempts the policy granted, never more. Under LimitedErrorCountRetryPolicy(n) that is at most n lines per operation, and the final give-up attempt is silent. The old version could emit n+1.

So the behaviour is unchanged from what is in the PR body — one WARNING per retried attempt — and I have removed the open question about powers-of-two sampling.

@v-pratap

Copy link
Copy Markdown
Contributor Author

Update: I re-ran this against the storage-testbench, and it found something that every unit test missed. Flagging it because it is security relevant and it extends an already-answered thread.

The gRPC upload id was being logged in full

In the earlier discussion about redacting the upload id, the shape under consideration was the resource name projects/<project>/buckets/<bucket>/<token>. My RetryLogUploadResource() handled that shape and returned anything unrecognised unchanged.

Against a real gRPC resumable upload, the session id turned out to be a bare opaque 64-character hex string with no structure at all, so it fell into the unchanged branch and the log published it whole:

[gcs-retry] UploadChunk: transient error on attempt 1,
  resource=d8bec938b31cdd41225a65f98da14d12ada93fa6ef3f5ccfbc8840d29b828391,
  status=UNAVAILABLE: ...

That is the entire bearer capability for the upload session. The default was fail-open for a credential, which is the worst way to be wrong.

Fix

RetryLogUploadResource() is now fail-closed: anything not structurally recognised is redacted whole, keeping only a short prefix so records from one upload can still be correlated. Same output against the testbench now:

[gcs-retry] UploadChunk: transient error on attempt 1,
  resource=7d7ab233...[redacted], status=UNAVAILABLE: ...

REST is unchanged and still keeps the useful part of the URL:

resource=http://localhost:9000/upload/storage/v1/b/retry-logging-demo-bucket/o
  ?uploadType=resumable&upload_id=a40d2c89...[redacted]

The header comment now explains why the default is fail-closed, so it does not get "helpfully" relaxed later. The unit test that asserted the pass-through behaviour has been deleted and replaced with tests for the opaque shapes, and UploadChunkLogsTransientFailures now uses a realistic resumable session URL and asserts that the bucket survives while the upload_id does not.

Full testbench matrix is in the PR description.

@v-pratap
v-pratap requested a review from kalragauri September 22, 2026 08:30
When GCS returns a transient error the client backs off and retries with
no output at all. From the application's point of view a read just takes
a long time. There is nothing to grep for, so an operator cannot tell a
slow network from a server that is shedding load, and the only way to
see the retries is to enable the `rawclient` tracing component and pay
for a full dump of every request and response.

This adds one WARNING record per retried attempt on the synchronous
paths, which cover both the JSON/REST and the gRPC transports (GCS gRPC
sync goes through `RestRetryLoop`, because `GenericStub` takes a
`RestContext&` regardless of transport).

    [gcs-retry] ReadObject/open: transient error on attempt 1,
    resource=my-bucket/my-object, status=UNAVAILABLE: ...

Call sites:

- `ConnectionImpl::ReadObject` -- an attempt failed to open the stream.
- `RetryObjectReadSource::Read` -- the stream broke mid-download and is
  being resumed from the last received byte.
- `ConnectionImpl::UploadChunk` -- an attempt to upload a chunk of a
  resumable upload failed.

Every record comes from the retry policy actually in force, never from a
second opinion about which errors are retryable. `ReadObject` wraps the
caller's policy in a `LoggingRetryPolicy` decorator and hands that to
`RestRetryLoop`; the other two sites already sit behind an `OnFailure()`
that returned true. So a line is published only when another attempt
really does follow. A permanent error, the final attempt of an exhausted
policy, and a non-idempotent request are all silent, because each is
reported to the caller through the returned `Status` instead. It also
means a caller who overrides the retry policy gets logging that matches
their policy, with no classification logic to drift out of step.

The severity is WARNING and the record is not gated behind a tracing
component. `StdClogBackend`, the default backend, drops everything below
FATAL, so this is invisible to a default OSS build; it becomes visible
to anyone who installs a `LogBackend`. A log that has to be turned on
first is not useful during the first occurrence of an incident, which is
exactly when it is needed.

Two details worth calling out:

- The status is flattened to a single line. GCS error bodies end with a
  newline, which would otherwise push the backend's `(file:line)` suffix
  onto a second line and make the record unusable with `grep`. It also
  removes a log-injection hazard, since the message is service-supplied.
- `UploadChunkRequest` only exposes the resumable session identifier,
  and the token in it is a bearer capability: anyone holding it can
  append to or finalize the upload for the lifetime of the session. Both
  transport forms are redacted to a short prefix -- the `upload_id`
  query parameter on REST, and the trailing token of
  `projects/<project>/buckets/<bucket>/<token>` on gRPC.

Also makes `LoggingStub::ReadObject` report the status on failure. It
cannot use `LogWrapper` because `unique_ptr<ObjectReadSource>` has no
`operator<<`, so it previously logged nothing about why the call failed.

Verified against the storage-testbench using its fault-injection API, in
addition to the unit tests.
@v-pratap
v-pratap merged commit 2269118 into googleapis:main Sep 22, 2026
68 checks passed

This branch was previously deployed

1 inactive deployment
false — 98cadb67 Deployed Sep 22, 2026 by v-pratap via Save PR ref #11916
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the Cloud Storage API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

⌂ 换网址