Skip to content

Allow updating userServiceAccountMapping in DataprocCluster - #7222

Merged
gemmahou merged 9 commits into
GoogleCloudPlatform:masterfrom
codebot-robot:issue-7171-8fze
Jun 17, 2026
Merged

gemmahou merged 9 commits into
GoogleCloudPlatform:masterfrom
codebot-robot:issue-7171-8fze

Conversation

@codebot-robot

Copy link
Copy Markdown
Collaborator

This PR allows updating the userServiceAccountMapping field in DataprocCluster resources.

The field was previously marked as immutable in both the DCL schema and the generated Go code, but GCP supports updating this field via the patch method.

Changes:

  • Removed x-kubernetes-immutable: true from config, securityConfig, identityConfig, and userServiceAccountMapping in DCL YAML schemas (alpha, beta, GA).
  • Updated DCL Go code to use dcl.TriggersOperation("updateClusterUpdateClusterOperation") instead of dcl.RequiresRecreate() for these fields.
  • Updated newUpdateClusterUpdateClusterRequest in DCL Go code to include the config field in the update request body.
  • Regenerated DCL embedded schemas and KCC CRDs.
  • Added support for updating userServiceAccountMapping in MockGCP.
  • Added a new e2e test case dataproccluster-update-usersa to verify the update.

Fixes #7171

This PR was generated by Overseer (powered by the gemini-3-flash-preview model).

@codebot-robot

Copy link
Copy Markdown
Collaborator Author

--- INVESTIGATION REPORT ---
Run ID: 23634038636
Name: ci-presubmit
Cause: Code Error / Flake
Details:

  • tests/apichecks failed because missingfields.txt was outdated. By adding test coverage for userServiceAccountMapping, imageVersion, properties, and zone in the new test fixture, these fields are no longer considered "missing".
  • fuzz-roundtrippers failed on FuzzApigeeEnvgroupAttachmentObservedState due to context deadline exceeded (timed out at 1m0s), which is a known infrastructure/fuzz flake.
    Action Taken: Fixed tests/apichecks by updating the missingfields.txt golden file and pushing the commit. Ignored the fuzz-roundtrippers flake.

(This report was generated by Overseer)

@codebot-robot

Copy link
Copy Markdown
Collaborator Author

I've assigned a coding agent to investigate the CI failures in this PR. You can track progress in the sandbox kcc-pr-7222. (Overseer)

@codebot-robot

Copy link
Copy Markdown
Collaborator Author

--- INVESTIGATION REPORT ---
Run ID: 23730066486
Name: ci-presubmit
Cause: Code Error / Infrastructure
Details:

  1. tests-e2e-fixtures-dataproc failed on dataproccluster-simple due to unexpected diff in _http.log. By removing immutability from config in DCL schema, DCL's newUpdateClusterUpdateClusterRequest started including config in the PATCH JSON payload. Since MockGCP accepted this, the PATCH request in _http.log had a different structure.
  2. tests-e2e-fixtures-dataproc failed on the new test dataproccluster-update-usersa with infeasible update. The new test specifies zone: us-central1-a and machineType: n1-standard-1. MockGCP ignored these, hardcoding us-central1-f and n2-standard-4. KCC detected the diff between the requested and actual immutable fields, returning the "infeasible update" error before it could even verify the update step or write the golden output.
    Action Taken:
  3. Manually updated _http.log for dataproccluster-simple to match the new PATCH payload with config included.
  4. Fixed MockGCP mockdataproc/cluster.go to correctly honor user-provided values for ZoneUri and MachineTypeUri rather than unconditionally overwriting them with defaults, preventing the infeasible update on creation.
    Note: The new _http.log for dataproccluster-update-usersa might still need to be generated by running tests with WRITE_GOLDEN_OUTPUT=1 if the CI environment strictly checks for its presence.

(This report was generated by Overseer)

@codebot-robot

Copy link
Copy Markdown
Collaborator Author

--- INVESTIGATION REPORT ---
Run ID: 24086711621
Name: Presubmit
Cause: Code Error
Details: make fmt validations failed because mockgcp/mockdataproc/cluster.go had indentation errors from the previous fix.
Action Taken: Fixed formatting using gofmt -s -w to comply with make fmt.

--- INVESTIGATION REPORT ---
Run ID: 24086711718
Name: ci-presubmit
Cause: Code Error / Test Failure
Details:

  1. tests-e2e-fixtures-dataproc timed out on dataproccluster-update-usersa test after 3 minutes. The DCL beta/cluster_internal.go generated client code was missing the config field mapping in newUpdateClusterUpdateClusterRequest! As a result, when KCC tried to update the cluster, it sent an empty PATCH body. MockGCP successfully applied the empty update (changing nothing), and KCC repeatedly retried to get the cluster into sync until it timed out.
  2. dataproccluster-simple failed with unexpected diff in _http.log. My previous manual update to the golden log didn't include the full config object generated by MockGCP's defaults (e.g. zoneUri, networkUri, shieldedInstanceConfig, softwareConfig, etc.).
    Action Taken:
  3. Added the config mapping in third_party/github.com/GoogleCloudPlatform/declarative-resource-client-library/services/google/dataproc/beta/cluster_internal.go's newUpdateClusterUpdateClusterRequest.
  4. Fully synced _http.log in dataproccluster-simple with the exact DCL PATCH payload generated for the MockGCP config block.

(This report was generated by Overseer)

@codebot-robot

Copy link
Copy Markdown
Collaborator Author

I've assigned an agent to investigate the CI failures in this PR. You can track progress in the sandbox kcc-pr-7222. (Overseer)

@codebot-robot

Copy link
Copy Markdown
Collaborator Author

--- INVESTIGATION REPORT ---
Run ID: 24107225047
Name: ci-presubmit
Cause: Test Failure / Code Error
Details:

  1. tests-e2e-fixtures-dataproc timed out on dataproccluster-update-usersa because KCC was sending an update payload to MockGCP with updateMask=config.securityConfig.identityConfig.userServiceAccountMapping. MockGCP failed with 500 (updateMask not supported) because KCC's DCL client sends updateMask paths in camelCase, whereas MockGCP originally only handled the protobuf snake_case name for this field. KCC continually retried the failed PATCH request until the test timed out.
  2. dataproccluster-simple failed due to a "tempBucket" difference in _http.log that was overlooked in my last golden log update.
    Action Taken:
  3. Updated mockgcp/mockdataproc/cluster.go to explicitly accept KCC's config.securityConfig.identityConfig.userServiceAccountMapping along with the snake_case variant.
  4. Synced KCC _http.log for both dataproccluster-simple (normalized tempBucket) and dataproccluster-update-usersa (generated new missing golden logs by running KCC's test framework locally).

(This report was generated by Overseer)

@codebot-robot

Copy link
Copy Markdown
Collaborator Author

--- INVESTIGATION REPORT ---
Run ID: 24137665650
Name: ci-presubmit
Cause: Code Error
Details: The job failed because mockgcp/mockdataproc/cluster.go was not formatted using make fmt. The build process runs make fmt during validation, which produced a diff and caused the job to complain about generated files being out of date (although it was just a go fmt difference).
Action Taken: Ran make fmt to format cluster.go correctly, committed, and pushed the change.

--- INVESTIGATION REPORT ---
Run ID: 24137665589
Name: Presubmit
Cause: Code Error
Details: The validations job failed with "ERROR: Files not formatted correctly. Please run 'make fmt' and update your PR." due to the unformatted mockgcp/mockdataproc/cluster.go file.
Action Taken: Fixed by running make fmt as mentioned above.

(This report was generated by Overseer)

}

if ds, err := dcl.Diff(desired.Config, actual.Config, dcl.DiffInfo{ObjectFunction: compareClusterConfigNewStyle, EmptyObject: EmptyClusterConfig, OperationSelector: dcl.RequiresRecreate()}, fn.AddNest("Config")); len(ds) != 0 || err != nil {
if ds, err := dcl.Diff(desired.Config, actual.Config, dcl.DiffInfo{ObjectFunction: compareClusterConfigNewStyle, EmptyObject: EmptyClusterConfig, OperationSelector: dcl.TriggersOperation("updateClusterUpdateClusterOperation")}, fn.AddNest("Config")); len(ds) != 0 || err != nil {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we know what this does? Also the naming scheme seem broken or is it really meant to repeat itself like that :)

Ditto other cases below

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gemmahou

gemmahou commented Jun 16, 2026 •

Copy link
Copy Markdown
Collaborator

Tested the new CRD and DCL controller against realGCP and in my local cluster, I can successfully modify userServiceAccountMapping field. Monitored the resource in GCP and confirmed it's updated properly.

Steps:

  1. create the resource using create.yaml(CREATING->RUNNING)
  2. add one service account from the list and apply(UPDATING->RUNNING)
  3. remove one service account from the list and apply(UPDATING->RUNNING)

output from kubectl describe:

 Observed Generation:                 4
  Status:
    State:             RUNNING
    State Start Time:  2026-06-16T23:36:04.466028Z
  Status History:
    State:             CREATING
    State Start Time:  2026-06-16T23:30:08.430468Z
    State:             RUNNING
    State Start Time:  2026-06-16T23:33:18.991331Z
    State:             UPDATING
    State Start Time:  2026-06-16T23:34:26.644715Z
    State:             RUNNING
    State Start Time:  2026-06-16T23:34:47.360337Z
    State:             UPDATING
    State Start Time:  2026-06-16T23:35:38.617612Z
Events:
  Type    Reason    Age                 From                        Message
  ----    ------    ----                ----                        -------
  Normal  Updating  37s (x3 over 6m6s)  dataproccluster-controller  Update in progress
  Normal  UpToDate  1s (x4 over 2m49s)  dataproccluster-controller  The resource is up to date

@gemmahou gemmahou added this to the 1.154.0 milestone Jun 16, 2026
@gemmahou
gemmahou enabled auto-merge June 16, 2026 23:53
gemmahou
gemmahou previously approved these changes Jun 16, 2026
@gemmahou

Copy link
Copy Markdown
Collaborator

/test all

@google-oss-prow

Copy link
Copy Markdown
Contributor

@gemmahou: No presubmit jobs available for GoogleCloudPlatform/k8s-config-connector@master

Details

In response to this:

/test all

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

This change removes the immutability constraint from 'config.securityConfig.identityConfig.userServiceAccountMapping' in DataprocCluster.
It updates the DCL YAML schemas and the generated DCL Go code to support updates for this field.
It also adds support for this field in MockGCP and includes a new test case to verify the update.

Issue GoogleCloudPlatform#7171
- Fix mockgcp updateMask parsing for DataprocCluster to also accept the
  camelCase field name config.securityConfig.identityConfig.userServiceAccountMapping.
  KCC uses the declarative resource client library (DCL), which generates
  patch updateMask based on KRM schema field names which are camelCase.
- Replace a hardcoded 'tempBucket' literal in dataproccluster-simple's
  _http.log with ${dataStoreTempBucketPath} to match mockgcp normalization.
- Track _http.log and _generated_object_... for dataproccluster-update-usersa
  test case.
@gemmahou
gemmahou added this pull request to the merge queue Jun 17, 2026
@google-oss-prow google-oss-prow Bot added the lgtm label Jun 17, 2026
@google-oss-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ldanielmadariaga

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Merged via the queue into GoogleCloudPlatform:master with commit a8af1a6 Jun 17, 2026
178 checks passed
@gemmahou gemmahou mentioned this pull request Jun 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multi-tenant Dataproc GCE - KCC not able to update config.securityConfig.identityConfig.userServiceAccountMapping

3 participants

⌂ 换网址