Repository navigation
Description
Bug Description
When running cloud-sql-proxy in a custom universe domain (e.g., s3nsapis.fr) with service account impersonation enabled (impersonate-service-account), the proxy fails during startup when attempting to refresh instance metadata, returning HTTP 401 ACCESS_TOKEN_TYPE_UNSUPPORTED.
An analysis of the error payload indicates that while the proxy correctly targets the custom universe domain for the Cloud SQL Admin API (sqladmin.s3nsapis.fr), the underlying client created for Service Account impersonation still defaults to the public Google IAM endpoint (iamcredentials.googleapis.com).
Environment
- Cloud SQL Proxy Version: v2.24.1
- OS: macOS / Linux
- Authentication Method: Custom Universe Domain Credentials / Workforce Identity Federation (WIF)
Configuration Used
instance-connection-name = "s3ns:sandbox-yohan-xxxx:u-france-east1:mydatabase"
universe-domain = "s3nsapis.fr"
impersonate-service-account = "my-app@sandbox-yohan-xxxx.s3ns.iam.gserviceaccount.com"
psc = trueSteps to Reproduce
1- Log in with a principal in a custom universe domain (e.g., s3nsapis.fr).
2- Run cloud-sql-proxy configured with both universe-domain and impersonate-service-account.
3- Observe the proxy initialization log.
Observed Behavior
The proxy fails to fetch instance metadata with the following error log:
2026/07/28 14:27:43 Impersonating service account with gcloud user credentials
2026/07/28 14:27:44 [s3ns:sandbox-yohan-09e8:u-france-east1:demo-replatforming] could not resolve instance version: failed to get instance: refresh error: failed to get instance metadata (connection name = "s3ns:sandbox-yohan-09e8:u-france-east1:demo-replatforming"): Get "https://sqladmin.s3nsapis.fr/sql/v1beta4/projects/s3ns%3Asandbox-yohan-09e8/instances/demo-replatforming/connectSettings?alt=json&prettyPrint=false": impersonate: status code 401: {
"error": {
"code": 401,
"message": "Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"status": "UNAUTHENTICATED",
"details": [
{
"@type": "type.googleapis.com/google.rpc.ErrorInfo",
"reason": "ACCESS_TOKEN_TYPE_UNSUPPORTED",
"metadata": {
"method": "google.iam.credentials.v1.IAMCredentials.GenerateAccessToken",
"service": "iamcredentials.googleapis.com"
}
}
]
}
}
Notice in the error metadata:
"service": "iamcredentials.googleapis.com" (Points to public Google Cloud instead of iamcredentials.s3nsapis.fr).
Expected Behavior
The universe-domain option should be propagated to the IAM Credentials client used by google.golang.org/api/impersonate during token generation. The proxy should call https://iamcredentials./... (e.g., iamcredentials.s3nsapis.fr) instead of iamcredentials.googleapis.com.
Root Cause Analysis
When universe-domain is configured, it is successfully applied to the Cloud SQL Admin API client (sqladmin.). However, when initializing the credentials for impersonate-service-account, the universe-domain configuration is not passed down to the impersonate package options (option.WithUniverseDomain(...)). Consequently, the impersonate client defaults to googleapis.com, which rejects authentication tokens issued by custom universe domains.
Activity
hessjcg commentedon Jul 29, 2026
Hi @ybelleguic, thank you for the detailed report! This will help us fix this long-standing issue connecting to a universe domains with service account impersonation.
yuhuayang-google commentedon Sep 29, 2026
Hi @ybelleguic, commit db8644c (PR #2547) added
option.WithUniverseDomain(c.UniverseDomain)toioptspassed toimpersonate.CredentialsTokenSourceininternal/proxy/proxy.go. Could you check if this issue is already resolved in v2.21.1+?ybelleguic commentedon Oct 1, 2026
hi @yuhuayang-google
I will check this tomorow ou next week, I'll keep you posted
ybelleguic commentedon Oct 5, 2026
I tested again using
v2.26.0with the following command:The proxy starts, but fails as soon as a connection is accepted and instance metadata is refreshed:
(connection name = "s3ns:sandbox-yohan-xxxx:u-france-east1:my-app"): Get "https://sqladmin.s3nsapis.fr/sql/v1beta4/projects/s3ns%3Asandbox-yohan-xxxx/instances/my-
app/connectSettings?alt=json&prettyPrint=false": impersonate: status code 404: {
"error": {
"code": 404,
"message": "Not found; Gaia id not found for email my-app@sandbox-yohan-xxxx.s3ns.iam.gserviceaccount.com",
"status": "NOT_FOUND"
}
}
Analysis
The error Gaia id not found for email my-app@sandbox-yohan-xxxx.s3ns.iam.gserviceaccount.com confirms that the impersonation request is still hitting the public Google IAM / Gaia
endpoint ( iamcredentials.googleapis.com ) instead of the custom universe domain endpoint ( iamcredentials.s3nsapis.fr ).
Since the S3NS service account only exists in the s3nsapis.fr universe domain, public Gaia returns HTTP 404.
It appears that --universe-domain is still not being propagated to the impersonate package options / credentials configuration when generating the impersonated token source.
LF3551 commentedon Nov 12, 2026
The universe domain does reach
impersonate.CredentialsTokenSourcesince #2547, butgoogle.golang.org/api/impersonateignores it for the request: the IAM Credentials endpoint is hardcoded tohttps://iamcredentials.googleapis.com(impersonate.go#L25, #L177), still the case in v0.313.0. That matches theGaia id not foundfrom the public IAM in the last report.The library side is tracked in googleapis/google-api-go-client#3739, with an open, not yet reviewed fix in googleapis/google-api-go-client#3741. I checked the options the proxy passes (
option.WithUniverseDomain("s3nsapis.fr")plus a token source) against a capturing HTTP client: v0.313.0 sendsgenerateAccessTokentoiamcredentials.googleapis.com, the #3741 branch sends it toiamcredentials.s3nsapis.fr, and the default universe is unchanged.So this issue should be fixed by #3741 plus a dependency bump here. The AlloyDB Auth Proxy builds its impersonation token source the same way, so it is affected too.