Repository navigation
ComposerEnvironment: Ensure the resource status is RUNNING before updating - #12365
Merged
gemmahou merged 3 commits intoAug 15, 2026
Merged
Conversation
gemmahou
marked this pull request as ready for review
August 13, 2026 17:11
gemmahou
force-pushed
the
composerenv-update
branch
from
August 13, 2026 17:14
b0c24a1 to
8582b53
Compare
gemmahou
commented
Aug 13, 2026
|
|
||
| --- | ||
|
|
||
| PATCH https://composer.googleapis.com/v1/projects/${projectId}/locations/us-central1/environments/composerenvironment-${uniqueId}?%24alt=json%3Benum-encoding%3Dint&updateMask=labels |
Collaborator
Author
There was a problem hiding this comment.
Repeated PATCH and DELETE calls are gone from the realGCP logs, which confirms the fix in the direct controller.
maqiuyujoyce
enabled auto-merge
August 15, 2026 00:11
gemmahou
force-pushed
the
composerenv-update
branch
from
August 15, 2026 00:18
c095e7c to
0eb0a01
Compare
gemmahou
force-pushed
the
composerenv-update
branch
from
August 15, 2026 00:23
0eb0a01 to
1b834e3
Compare
maqiuyujoyce
previously approved these changes
Aug 15, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to a conflict with the base branch
Aug 15, 2026
gemmahou
force-pushed
the
composerenv-update
branch
from
August 15, 2026 01:28
1b834e3 to
58e06c3
Compare
gemmahou
force-pushed
the
composerenv-update
branch
from
August 15, 2026 01:30
58e06c3 to
2024ebf
Compare
gemmahou
enabled auto-merge
August 15, 2026 01:41
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: maqiuyujoyce The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Merged
via the queue into
GoogleCloudPlatform:master
with commit Aug 15, 2026
e153339
246 checks passed
maqiuyujoyce
pushed a commit
that referenced
this pull request
Aug 18, 2026
…ating (#12365) ### BRIEF Change description <!-- Describe what this pull request does. * If your pull request is to address an open issue, indicate it by specifying the issue number: * If your pull request fixes an issue which has not been filed, please file the issue and put the number here. For example: "Fixes #858" --> Updating a ComposerEnvironment takes approximately 10 minutes to complete. While the resource status remains UPDATING, modifications are not reflected in the actual state. The diff between the desired and actual states triggers repeated Update API calls until the environment transitions to RUNNING and the changes are successfully applied. Skip the update of a ComposerEnvironment when the state of the underlying GCP Composer environment is not RUNNING / is UPDATING, to avoid overload the API when it's still working on the environment. #### WHY do we need this change? #### Special notes for your reviewer: #### Does this PR add something which needs to be 'release noted'? <!-- If no, just write "NONE" in the release-note block below. If yes, a release note is required: Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required". --> ```release-note Bug fix ComposerEnvironment: Skip the update of a ComposerEnvironment when the state of the underlying GCP Composer environment is not RUNNING. ``` - [ ] Reviewer reviewed release note. #### Additional documentation e.g., references, usage docs, etc.: <!-- This section can be blank if this pull request does not require any additional documentation. When adding links which point to resources within git repositories, like usage documentation, please reference a specific commit and avoid linking directly to the master branch. This ensures that links reference a specific point in time, rather than a document that may change over time. See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files Please use the following format for linking documentation: - [Usage]: <link> - [Other doc]: <link> --> ```docs ``` #### Intended Milestone Please indicate the intended milestone. - [ ] Reviewer tagged PR with the actual milestone. ### Tests you have done <!-- Make sure you have run "make ready-pr" to run required tests and ensure this PR is ready to review. Also if possible, share a bit more on the tests you have done. For example if you have updated the pubsubtopic sample, you can share the test logs from running the test case locally. go test -v -tags=integration ./config/tests/samples/create -test.run TestAll -run-tests pubsubtopic --> - [ ] Run `make ready-pr` to ensure this PR is ready for review. - [ ] Perform necessary E2E testing for changed resources.
Merged
Merged
daedalus-agent-bot
pushed a commit
to daedalus-agent-bot/k8s-config-connector
that referenced
this pull request
Sep 23, 2026
…nt (GoogleCloudPlatform#12795) Fix multiple reconciliation issues in the ComposerEnvironment direct controller that caused false drift, repeated 400 rejections, and quota exhaustion. ### BRIEF Change description This PR addresses three reconciliation and diffing issues in the `ComposerEnvironment` direct controller: 1. **Empty Map Diffing**: Switched from `reflect.DeepEqual` to `maps.Equal` for map fields (`labels`, `env_variables`, `airflow_config_overrides`, `pypi_packages`). This treats unmarshaled empty KRM maps (`{}`) as equivalent to `nil` proto maps, eliminating false drift while still properly detecting when existing server values are cleared. 2. **Granular `WorkloadsConfig` Defaulting**: Expanded `computedFieldPaths` in `composerenvironment_defaults.go` to include leaf fields across all workload components (`Scheduler`, `DagProcessor`, `Triggerer`, `WebServer`, `Worker`). This ensures partial updates (e.g., setting only `triggerer.count`) inherit live server values for omitted sibling fields rather than zeroing them out. 3. **Scheduled Snapshots Disablement**: Updated the `scheduled_snapshots_config` updater to preserve live server values for `snapshotLocation`, `snapshotCreationSchedule`, and `timeZone` when `enabled: false` is set with omitted fields. This prevents second-pass false drift against the server-retained configuration and ignores redundant disable requests when already disabled. Fixes b/543582925 #### WHY do we need this change? * **Empty map specifications (`envVariables: {}`, `labels: {}`)**: In Go, `reflect.DeepEqual(map[string]string{}, nil)` evaluates to `false`. Because the GCP API returns unset maps as `nil`, KCC detects continuous false drift and sends a no-op PATCH every 2 minutes. Cloud Composer rejects this with HTTP 400 (`"Update request does not result in any change"`), entering an infinite reconcile loop that rapidly exhausts daily write API quota. * **Partial workload updates**: `computedFieldPaths` previously tracked workload components only at the top struct level. Setting a single subfield marked the parent struct as present, skipping dynamic default population for omitted sibling fields. In proto3, these defaulted to `0` (e.g., `cpu: 0`), which Cloud Composer rejects with HTTP 400. * **Disabling scheduled snapshots**: When snapshots are disabled in GCP, the API retains the existing location and schedule in the live environment. If a manifest specifies `enabled: false` without repeating the location string, KCC diffs the omitted fields against the live retained values, triggering recurring no-op PATCH calls and 400 errors. #### Special notes for your reviewer: * Root cause analysis and proposed fixes were discussed with and approved by @maqiuyujoyce in b/543582925#comment49. * **Scope of `maps.Equal`**: As noted in b/543582925#comment52, replacing `reflect.DeepEqual` with `maps.Equal` is strictly scoped to `ComposerEnvironment` in this PR. * **WorkloadsConfig leaf defaulting**: Inheriting omitted leaf fields from `actualPb` prevents 0-value rejections on partial updates without requiring fine-grained leaf-level update masks, which Cloud Composer API rejects when multiple paths are combined. #### Does this PR add something which needs to be 'release noted'? ```release-note Fix infinite reconciliation loops, false drift, and HTTP 400 errors on ComposerEnvironment when using empty map fields (`envVariables: {}`, `labels: {}`), partial `workloadsConfig` blocks, or disabling `scheduledSnapshotsConfig`. ``` - [ ] Reviewer reviewed release note. #### Additional documentation e.g., references, usage docs, etc.: ```docs - Related PRs: GoogleCloudPlatform#12364, GoogleCloudPlatform#12365 ``` #### Intended Milestone 1.158.0 - [ ] Reviewer tagged PR with the actual milestone. ### Tests you have done - [X] Run `make ready-pr` to ensure this PR is ready for review. - [X] Perform necessary E2E testing for changed resources. All three scenarios (`empty values "{}"`, `scheduledSnapshotsConfig.enabled: false`, `partial workload config`) were validated against a real KCC cluster / Composer env. The "fixed" version successfully updated the Composer env. ``` $ cat composer-snapshots-enabled.yaml apiVersion: composer.cnrm.cloud.google.com/v1beta1 kind: ComposerEnvironment metadata: name: composer-repro-snapshots namespace: rapid-labs-gcp spec: projectRef: external: testproject location: europe-west3 config: environmentSize: ENVIRONMENT_SIZE_SMALL nodeConfig: serviceAccountRef: external: kcc-test-sa@testproject.iam.gserviceaccount.com softwareConfig: imageVersion: composer-3-airflow-2.10.5-build.52 recoveryConfig: scheduledSnapshotsConfig: enabled: true snapshotLocation: gs://tf-solution-xyz/snapshots snapshotCreationSchedule: "0 5 * * *" timeZone: "UTC" $ diff composer-snapshots-enabled.yaml composer-snapshots-disabled.yaml 19,22c19 < enabled: true < snapshotLocation: gs://tf-solution-xyz/snapshots < snapshotCreationSchedule: "0 5 * * *" < timeZone: "UTC" --- > enabled: false $ diff composer-snapshots-enabled.yaml composer-test-empty-maps.yaml 16a17 > envVariables: {} $ diff composer-snapshots-enabled.yaml composer-test-partial-workloads.yaml 16a17,19 > workloadsConfig: > scheduler: > count: 2 ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BRIEF Change description
Updating a ComposerEnvironment takes approximately 10 minutes to complete. While the resource status remains UPDATING, modifications are not reflected in the actual state. The diff between the desired and actual states triggers repeated Update API calls until the environment transitions to RUNNING and the changes are successfully applied.
Skip the update of a ComposerEnvironment when the state of the underlying GCP Composer environment is not RUNNING / is UPDATING, to avoid overload the API when it's still working on the environment.
WHY do we need this change?
Special notes for your reviewer:
Does this PR add something which needs to be 'release noted'?
Additional documentation e.g., references, usage docs, etc.:
Intended Milestone
Please indicate the intended milestone.
Tests you have done
make ready-prto ensure this PR is ready for review.