Skip to content

[BUG] universe-domain is not propagated to IAMCredentials client when using impersonate-service-account #2640

Open

Description

@ybelleguic

Bug Description

When running cloud-sql-proxy in a custom universe domain (e.g., s3nsapis.fr) with service account impersonation enabled (impersonate-service-account), the proxy fails during startup when attempting to refresh instance metadata, returning HTTP 401 ACCESS_TOKEN_TYPE_UNSUPPORTED.

An analysis of the error payload indicates that while the proxy correctly targets the custom universe domain for the Cloud SQL Admin API (sqladmin.s3nsapis.fr), the underlying client created for Service Account impersonation still defaults to the public Google IAM endpoint (iamcredentials.googleapis.com).

Environment

  • Cloud SQL Proxy Version: v2.24.1
  • OS: macOS / Linux
  • Authentication Method: Custom Universe Domain Credentials / Workforce Identity Federation (WIF)

Configuration Used

instance-connection-name = "s3ns:sandbox-yohan-xxxx:u-france-east1:mydatabase"
universe-domain = "s3nsapis.fr"
impersonate-service-account = "my-app@sandbox-yohan-xxxx.s3ns.iam.gserviceaccount.com"
psc = true

Steps to Reproduce

1- Log in with a principal in a custom universe domain (e.g., s3nsapis.fr).
2- Run cloud-sql-proxy configured with both universe-domain and impersonate-service-account.
3- Observe the proxy initialization log.

Observed Behavior

The proxy fails to fetch instance metadata with the following error log:

2026/07/28 14:27:43 Impersonating service account with gcloud user credentials
2026/07/28 14:27:44 [s3ns:sandbox-yohan-09e8:u-france-east1:demo-replatforming] could not resolve instance version: failed to get instance: refresh error: failed to get instance metadata (connection name = "s3ns:sandbox-yohan-09e8:u-france-east1:demo-replatforming"): Get "https://sqladmin.s3nsapis.fr/sql/v1beta4/projects/s3ns%3Asandbox-yohan-09e8/instances/demo-replatforming/connectSettings?alt=json&prettyPrint=false": impersonate: status code 401: {
  "error": {
    "code": 401,
    "message": "Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
    "status": "UNAUTHENTICATED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "ACCESS_TOKEN_TYPE_UNSUPPORTED",
        "metadata": {
          "method": "google.iam.credentials.v1.IAMCredentials.GenerateAccessToken",
          "service": "iamcredentials.googleapis.com"
        }
      }
    ]
  }
}

Notice in the error metadata:

"service": "iamcredentials.googleapis.com" (Points to public Google Cloud instead of iamcredentials.s3nsapis.fr).

Expected Behavior

The universe-domain option should be propagated to the IAM Credentials client used by google.golang.org/api/impersonate during token generation. The proxy should call https://iamcredentials./... (e.g., iamcredentials.s3nsapis.fr) instead of iamcredentials.googleapis.com.

Root Cause Analysis

When universe-domain is configured, it is successfully applied to the Cloud SQL Admin API client (sqladmin.). However, when initializing the credentials for impersonate-service-account, the universe-domain configuration is not passed down to the impersonate package options (option.WithUniverseDomain(...)). Consequently, the impersonate client defaults to googleapis.com, which rejects authentication tokens issued by custom universe domains.

Activity

  1. added
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    on Jul 28, 2026
  2. hessjcg commented on Jul 29, 2026

    @hessjcg
    Member

    Hi @ybelleguic, thank you for the detailed report! This will help us fix this long-standing issue connecting to a universe domains with service account impersonation.

  3. yuhuayang-google commented on Sep 29, 2026

    @yuhuayang-google

    Hi @ybelleguic, commit db8644c (PR #2547) added option.WithUniverseDomain(c.UniverseDomain) to iopts passed to impersonate.CredentialsTokenSource in internal/proxy/proxy.go. Could you check if this issue is already resolved in v2.21.1+?

  4. ybelleguic commented on Oct 1, 2026

    @ybelleguic
    Author

    hi @yuhuayang-google
    I will check this tomorow ou next week, I'll keep you posted

  5. ybelleguic commented on Oct 5, 2026

    @ybelleguic
    Author

    I tested again using v2.26.0 with the following command:

    ```bash
    ./cloud-sql-proxy s3ns:sandbox-yohan-xxx:u-france-east1:my-app \
      --port 5432 \
      --impersonate-service-account=my-app@sandbox-yohan-xxxx.s3ns.iam.gserviceaccount.com \
      --psc \
      --universe-domain=s3nsapis.fr
    

    The proxy starts, but fails as soon as a connection is accepted and instance metadata is refreshed:

    2026/10/05 17:17:41 Impersonating service account with Application Default Credentials
    2026/10/05 17:17:41 [s3ns:sandbox-yohan-xxxx:u-france-east1:my-app] Listening on 127.0.0.1:5432
    2026/10/05 17:17:41 The proxy has started successfully and is ready for new connections!
    2026/10/05 17:17:46 [s3ns:sandbox-yohan-xxxx:u-france-east1:my-app] Accepted connection from 127.0.0.1:34596
    2026/10/05 17:17:46 [s3ns:sandbox-yohan-xxxx:u-france-east1:my-app] failed to connect to instance: failed to get instance: refresh error: failed to get instance metadata
    

    (connection name = "s3ns:sandbox-yohan-xxxx:u-france-east1:my-app"): Get "https://sqladmin.s3nsapis.fr/sql/v1beta4/projects/s3ns%3Asandbox-yohan-xxxx/instances/my-
    app/connectSettings?alt=json&prettyPrint=false": impersonate: status code 404: {
    "error": {
    "code": 404,
    "message": "Not found; Gaia id not found for email my-app@sandbox-yohan-xxxx.s3ns.iam.gserviceaccount.com",
    "status": "NOT_FOUND"
    }
    }

    Analysis

    The error Gaia id not found for email my-app@sandbox-yohan-xxxx.s3ns.iam.gserviceaccount.com confirms that the impersonation request is still hitting the public Google IAM / Gaia
    endpoint ( iamcredentials.googleapis.com ) instead of the custom universe domain endpoint ( iamcredentials.s3nsapis.fr ).

    Since the S3NS service account only exists in the s3nsapis.fr universe domain, public Gaia returns HTTP 404.

    It appears that --universe-domain is still not being propagated to the impersonate package options / credentials configuration when generating the impersonated token source.

  6. LF3551 commented on Nov 12, 2026

    @LF3551

    The universe domain does reach impersonate.CredentialsTokenSource since #2547, but google.golang.org/api/impersonate ignores it for the request: the IAM Credentials endpoint is hardcoded to https://iamcredentials.googleapis.com (impersonate.go#L25, #L177), still the case in v0.313.0. That matches the Gaia id not found from the public IAM in the last report.

    The library side is tracked in googleapis/google-api-go-client#3739, with an open, not yet reviewed fix in googleapis/google-api-go-client#3741. I checked the options the proxy passes (option.WithUniverseDomain("s3nsapis.fr") plus a token source) against a capturing HTTP client: v0.313.0 sends generateAccessToken to iamcredentials.googleapis.com, the #3741 branch sends it to iamcredentials.s3nsapis.fr, and the default universe is unchanged.

    So this issue should be fixed by #3741 plus a dependency bump here. The AlloyDB Auth Proxy builds its impersonation token source the same way, so it is affected too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ 换网址