Skip to content

Container Cluster stuck in non-ready state because channel update rejected  #194

Closed

Description

@jlewi

Describe the bug
A clear and concise description of what the bug is.

ConfigConnector Version
Run the following command to get the current ConfigConnector version

kubectl get ns cnrm-system -o jsonpath='{.metadata.annotations.cnrm\.cloud\.google\.com/version}' 

cnrm.cloud.google.com/version: 1.9.1

To Reproduce
Steps to reproduce the behavior:

  1. Create a ContainerCluster CNRM resource setting the release channel for the cluster to get
    the status

  2. Apply the resource to create the cluster

  3. Reapply the resource

  4. Container cluster reports

      - lastTransitionTime: "2020-05-29T00:15:38Z"
       message: 'Update call failed: the desired mutation for the following field(s)
         is invalid: [releaseChannel.0.Channel]'
       reason: UpdateFailed
       status: "False"
       type: Ready
    

The release channel shouldn't be changing.

I suspect this an issue in the update logic since there are some restrictions about mutations to
release channel
https://cloud.google.com/kubernetes-engine/docs/concepts/release-channels

YAML snippets:

apiVersion: container.cnrm.cloud.google.com/v1beta1
kind: ContainerCluster
metadata:
  clusterName: gcp-private-dev/us-central1/jl-0601
  labels:
    kf-name: gcp-private-0527
    mesh_id: gcp-private-dev_us-central1_jl-0601
  name: jl-0601
  namespace: gcp-private-dev
spec:
  clusterAutoscaling:
    autoProvisioningDefaults:
      oauthScopes:
      - https://www.googleapis.com/auth/logging.write
      - https://www.googleapis.com/auth/monitoring
      - https://www.googleapis.com/auth/devstorage.read_only
      serviceAccountRef:
        name: jl-0601-vm
    enabled: true
    resourceLimits:
    - maximum: 128
      resourceType: cpu
    - maximum: 2000
      resourceType: memory
    - maximum: 16
      resourceType: nvidia-tesla-k80
  initialNodeCount: 2
  ipAllocationPolicy:
    clusterSecondaryRangeName: pods
    createSubnetwork: false
    servicesSecondaryRangeName: services
    useIpAliases: true
  location: us-central1
  loggingService: logging.googleapis.com/kubernetes
  monitoringService: monitoring.googleapis.com/kubernetes
  networkRef:
    name: jl-0601
  nodeConfig:
    machineType: n1-standard-8
    metadata:
      disable-legacy-endpoints: "true"
    oauthScopes:
    - https://www.googleapis.com/auth/logging.write
    - https://www.googleapis.com/auth/monitoring
    - https://www.googleapis.com/auth/devstorage.read_only
    serviceAccountRef:
      name: jl-0601-vm
    workloadMetadataConfig:
      nodeMetadata: GKE_METADATA_SERVER
  privateClusterConfig:
    enablePrivateEndpoint: false
    enablePrivateNodes: true
    masterIpv4CidrBlock: 172.16.0.32/28
  releaseChannel:
    channel: stable
  subnetworkRef:
    name: jl-0601
  workloadIdentityConfig:
    identityNamespace: gcp-private-dev.svc.id.goog

Activity

  1. added
    bugSomething isn't working
    on Jun 1, 2020
  2. kibbles-n-bytes commented on Jun 3, 2020

    @kibbles-n-bytes
    Contributor

    Hey @jlewi , the underlying API's canonical form for the release channel is all uppercase: https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#channel

    Curiously, the underlying API accepts the lowercase form as well, but further GET requests return the uppercase form, which we currently don't suppress a diff for. If you create your cluster with channel: STABLE instead, you should see this issue go away.

  3. jlewi commented on Jun 3, 2020

    @jlewi
    Author

    Thanks that's super helpful I will fix that on our end.

    Feel free to close this issue unless you want to leave open to continue to track handling the diff more gracefully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ 换网址