Skip to content

ComposerEnvironment: Ensure the resource status is RUNNING before updating - #12365

Merged
gemmahou merged 3 commits into
GoogleCloudPlatform:masterfrom
gemmahou:composerenv-update
Aug 15, 2026
Merged

gemmahou merged 3 commits into
GoogleCloudPlatform:masterfrom
gemmahou:composerenv-update

Conversation

@gemmahou

@gemmahou gemmahou commented Aug 13, 2026 •

Copy link
Copy Markdown
Collaborator

BRIEF Change description

Updating a ComposerEnvironment takes approximately 10 minutes to complete. While the resource status remains UPDATING, modifications are not reflected in the actual state. The diff between the desired and actual states triggers repeated Update API calls until the environment transitions to RUNNING and the changes are successfully applied.

Skip the update of a ComposerEnvironment when the state of the underlying GCP Composer environment is not RUNNING / is UPDATING, to avoid overload the API when it's still working on the environment.

WHY do we need this change?

Special notes for your reviewer:

Does this PR add something which needs to be 'release noted'?

Bug fix
ComposerEnvironment: Skip the update of a ComposerEnvironment when the state of the underlying GCP Composer environment is not RUNNING.
  • Reviewer reviewed release note.

Additional documentation e.g., references, usage docs, etc.:


Intended Milestone

Please indicate the intended milestone.

  • Reviewer tagged PR with the actual milestone.

Tests you have done

  • Run make ready-pr to ensure this PR is ready for review.
  • Perform necessary E2E testing for changed resources.

@gemmahou gemmahou changed the title Collect realGCP logs ComposerEnvironment: Ensure the resource status is RUNNING before updating Aug 13, 2026
@gemmahou
gemmahou marked this pull request as ready for review August 13, 2026 17:11
@gemmahou
gemmahou force-pushed the composerenv-update branch from b0c24a1 to 8582b53 Compare August 13, 2026 17:14

---

PATCH https://composer.googleapis.com/v1/projects/${projectId}/locations/us-central1/environments/composerenvironment-${uniqueId}?%24alt=json%3Benum-encoding%3Dint&updateMask=labels

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Repeated PATCH and DELETE calls are gone from the realGCP logs, which confirms the fix in the direct controller.

maqiuyujoyce
maqiuyujoyce previously approved these changes Aug 15, 2026

@maqiuyujoyce maqiuyujoyce left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

maqiuyujoyce
maqiuyujoyce previously approved these changes Aug 15, 2026
@google-oss-prow google-oss-prow Bot added the lgtm label Aug 15, 2026
@maqiuyujoyce
maqiuyujoyce added this pull request to the merge queue Aug 15, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Aug 15, 2026

@maqiuyujoyce maqiuyujoyce left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@gemmahou
gemmahou added this pull request to the merge queue Aug 15, 2026
@google-oss-prow google-oss-prow Bot added the lgtm label Aug 15, 2026
@google-oss-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: maqiuyujoyce

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Merged via the queue into GoogleCloudPlatform:master with commit e153339 Aug 15, 2026
246 checks passed
@gemmahou
gemmahou deleted the composerenv-update branch August 15, 2026 02:12
maqiuyujoyce pushed a commit that referenced this pull request Aug 18, 2026
…ating (#12365)

### BRIEF Change description

<!--
Describe what this pull request does.

* If your pull request is to address an open issue, indicate it by
specifying the
issue number:
* If your pull request fixes an issue which has not been filed, please
file the
issue and put the number here.

For example: "Fixes #858"
-->
Updating a ComposerEnvironment takes approximately 10 minutes to
complete. While the resource status remains UPDATING, modifications are
not reflected in the actual state. The diff between the desired and
actual states triggers repeated Update API calls until the environment
transitions to RUNNING and the changes are successfully applied.

Skip the update of a ComposerEnvironment when the state of the
underlying GCP Composer environment is not RUNNING / is UPDATING, to
avoid overload the API when it's still working on the environment.

#### WHY do we need this change?

#### Special notes for your reviewer:

#### Does this PR add something which needs to be 'release noted'?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires
additional action from users switching to the new release, include the
string "action required".
-->
```release-note
Bug fix
ComposerEnvironment: Skip the update of a ComposerEnvironment when the state of the underlying GCP Composer environment is not RUNNING.
```

- [ ] Reviewer reviewed release note.

#### Additional documentation e.g., references, usage docs, etc.:

<!--
This section can be blank if this pull request does not require any
additional documentation.

When adding links which point to resources within git repositories, like
usage documentation, please reference a specific commit and avoid
linking directly to the master branch. This ensures that links reference
a
specific point in time, rather than a document that may change over
time.

See here for guidance on getting permanent links to files:
https://help.github.com/en/articles/getting-permanent-links-to-files

Please use the following format for linking documentation:
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs

```

#### Intended Milestone

Please indicate the intended milestone.
- [ ] Reviewer tagged PR with the actual milestone.

### Tests you have done

<!--

Make sure you have run "make ready-pr" to run required tests and ensure
this PR is ready to review.

Also if possible, share a bit more on the tests you have done.

For example if you have updated the pubsubtopic sample, you can share
the test logs from running the test case locally.

go test -v -tags=integration ./config/tests/samples/create -test.run
TestAll -run-tests pubsubtopic

-->

- [ ] Run `make ready-pr` to ensure this PR is ready for review.
- [ ] Perform necessary E2E testing for changed resources.
@barney-s barney-s mentioned this pull request Sep 1, 2026
daedalus-agent-bot pushed a commit to daedalus-agent-bot/k8s-config-connector that referenced this pull request Sep 23, 2026
…nt (GoogleCloudPlatform#12795)

Fix multiple reconciliation issues in the ComposerEnvironment direct
controller that caused false drift, repeated 400 rejections, and quota
exhaustion.

### BRIEF Change description

This PR addresses three reconciliation and diffing issues in the
`ComposerEnvironment` direct controller:

1. **Empty Map Diffing**: Switched from `reflect.DeepEqual` to
`maps.Equal` for map fields (`labels`, `env_variables`,
`airflow_config_overrides`, `pypi_packages`). This treats unmarshaled
empty KRM maps (`{}`) as equivalent to `nil` proto maps, eliminating
false drift while still properly detecting when existing server values
are cleared.
2. **Granular `WorkloadsConfig` Defaulting**: Expanded
`computedFieldPaths` in `composerenvironment_defaults.go` to include
leaf fields across all workload components (`Scheduler`, `DagProcessor`,
`Triggerer`, `WebServer`, `Worker`). This ensures partial updates (e.g.,
setting only `triggerer.count`) inherit live server values for omitted
sibling fields rather than zeroing them out.
3. **Scheduled Snapshots Disablement**: Updated the
`scheduled_snapshots_config` updater to preserve live server values for
`snapshotLocation`, `snapshotCreationSchedule`, and `timeZone` when
`enabled: false` is set with omitted fields. This prevents second-pass
false drift against the server-retained configuration and ignores
redundant disable requests when already disabled.

Fixes b/543582925

#### WHY do we need this change?

* **Empty map specifications (`envVariables: {}`, `labels: {}`)**: In
Go, `reflect.DeepEqual(map[string]string{}, nil)` evaluates to `false`.
Because the GCP API returns unset maps as `nil`, KCC detects continuous
false drift and sends a no-op PATCH every 2 minutes. Cloud Composer
rejects this with HTTP 400 (`"Update request does not result in any
change"`), entering an infinite reconcile loop that rapidly exhausts
daily write API quota.
* **Partial workload updates**: `computedFieldPaths` previously tracked
workload components only at the top struct level. Setting a single
subfield marked the parent struct as present, skipping dynamic default
population for omitted sibling fields. In proto3, these defaulted to `0`
(e.g., `cpu: 0`), which Cloud Composer rejects with HTTP 400.
* **Disabling scheduled snapshots**: When snapshots are disabled in GCP,
the API retains the existing location and schedule in the live
environment. If a manifest specifies `enabled: false` without repeating
the location string, KCC diffs the omitted fields against the live
retained values, triggering recurring no-op PATCH calls and 400 errors.

#### Special notes for your reviewer:

* Root cause analysis and proposed fixes were discussed with and
approved by @maqiuyujoyce in b/543582925#comment49.
* **Scope of `maps.Equal`**: As noted in b/543582925#comment52,
replacing `reflect.DeepEqual` with `maps.Equal` is strictly scoped to
`ComposerEnvironment` in this PR.
* **WorkloadsConfig leaf defaulting**: Inheriting omitted leaf fields
from `actualPb` prevents 0-value rejections on partial updates without
requiring fine-grained leaf-level update masks, which Cloud Composer API
rejects when multiple paths are combined.

#### Does this PR add something which needs to be 'release noted'?

```release-note
Fix infinite reconciliation loops, false drift, and HTTP 400 errors on ComposerEnvironment when using empty map fields (`envVariables: {}`, `labels: {}`), partial `workloadsConfig` blocks, or disabling `scheduledSnapshotsConfig`.
```

- [ ] Reviewer reviewed release note.

#### Additional documentation e.g., references, usage docs, etc.:

```docs
- Related PRs: GoogleCloudPlatform#12364, GoogleCloudPlatform#12365
```

#### Intended Milestone

1.158.0

- [ ] Reviewer tagged PR with the actual milestone.

### Tests you have done

- [X] Run `make ready-pr` to ensure this PR is ready for review.
- [X] Perform necessary E2E testing for changed resources.

All three scenarios (`empty values "{}"`,
`scheduledSnapshotsConfig.enabled: false`, `partial workload config`)
were validated against a real KCC cluster / Composer env. The "fixed"
version successfully updated the Composer env.

```
$ cat composer-snapshots-enabled.yaml
apiVersion: composer.cnrm.cloud.google.com/v1beta1
kind: ComposerEnvironment
metadata:
  name: composer-repro-snapshots
  namespace: rapid-labs-gcp
spec:
  projectRef:
    external: testproject
  location: europe-west3
  config:
    environmentSize: ENVIRONMENT_SIZE_SMALL
    nodeConfig:
      serviceAccountRef:
        external: kcc-test-sa@testproject.iam.gserviceaccount.com
    softwareConfig:
      imageVersion: composer-3-airflow-2.10.5-build.52
    recoveryConfig:
      scheduledSnapshotsConfig:
        enabled: true
        snapshotLocation: gs://tf-solution-xyz/snapshots
        snapshotCreationSchedule: "0 5 * * *"
        timeZone: "UTC"

 $ diff composer-snapshots-enabled.yaml composer-snapshots-disabled.yaml
19,22c19
<         enabled: true
<         snapshotLocation: gs://tf-solution-xyz/snapshots
<         snapshotCreationSchedule: "0 5 * * *"
<         timeZone: "UTC"
---
>         enabled: false

$ diff composer-snapshots-enabled.yaml composer-test-empty-maps.yaml
16a17
>       envVariables: {}

$ diff composer-snapshots-enabled.yaml composer-test-partial-workloads.yaml
16a17,19
>     workloadsConfig:
>       scheduler:
>         count: 2
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

⌂ 换网址