You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I need help understanding ContainerCluster. It seems like it's updating itself every 10th minute. This leads to our ContainerNodePool's sends out warnings:
reference ContainerCluster {{REDACTED}} is not ready
Additional Diagnostic Information
Kubernetes Cluster Version
Client Version: v1.19.2
Server Version: v1.20.10-gke.1600
Hello! As a quick clarification of what @mbzomowski was saying: although it is true that there will be an API GET every 10 minutes to ensure that the resource is up to date, it is not necessarily true that the update will occur every 10 minutes: only in the case where there was a change that needed to be applied.
So update requests every 10 minutes implies that there is something up with the resource: either it is not updating properly, or perhaps some erroneous diff detection on the part of Config Connector.
@travisrandolph-bestbuy can you send us a redacted version of the resource so we can see if we can reproduce locally? The output you sent looks to be logs from the controller, which can be helpful for diagnostics but doesn't help us spot issues like whether label leasing is on or reproducing locally.
Redacted output from a kubebctl get -o yaml would be perfect.
Hi @travisrandolph-bestbuy , sorry for the delayed response. I was trying to create a GKE cluster with the configuration you provided (except for enableBinaryAuthorization: true due to some issues that might require further digging), but was not able to reproduce this issue. The created cluster was not updated regularly.
On the other hand, I noticed that if there is any update to the cluster, there should be a corresponding log for the update operation. Could you check the GKE operation logs associated with the cluster to get more information about what has been updated and whether the changes in the operations match the yaml?
An example query to search for GKE operations in Logs Explorer would be:
Hi @travisrandolph-bestbuy, I think this issue is probably being caused by spec.releaseChannel.channel: stable. Can you try setting the field to STABLE instead?
Officially, the field only recognizes all-caps values as per the field's description in the docs. I can file a bug to allow for all-lower-case values as well to avoid this issue in the future, but we probably won't be able to prioritize this improvement for a while if the all-caps values do work as intended.
If using STABLE is not enough to fix the issue, it's possible that the diff is coming from one of the {{REDACTED}} values. In that case, I would try comparing your K8s resource state against the output of gcloud container clusters describe and look for any differences. This is, of course, not a perfect way to debug the issue since the two object models are different, but they are similar enough that you can sometimes use them to look for unintended diffs.
@jcanseco Updating the release channel to use upper case resolved some of our clusters. One of the logs looks like it was trying to update the maintenance window. After looking into it we were leaving the Zulu time specification off the end of the start and end times. Once I updated my configs to '2019-09-02T15:00:00Z', it stopped trying to update. After these changes most of our clusters are fine now. The last problem seems to be with 'masterAuthorizedNetworksConfig'. I've added the mismatch in the second code box below. We can't add the boolean, so I'm not sure we can do anything to fix that.
Hey @travisrandolph-bestbuy, I'm glad to hear that most of your clusters are no longer seeing the issue.
Re: masterAuthorizedNetworksConfig.enabled: this is actually one case where the "diff against gcloud output" approach doesn't work perfectly since KCC hides the enabled flag from the user (KCC sets it to true as long as masterAuthorizedNetworksConfig is set, and sets it to false otherwise).
Would you be willing to share the kubectl get -o yaml and gcloud container clusters describe outputs of the last problematic cluster(s)? We can try to take a look and help check for unintended diffs.
Activity
mbzomowski commentedon Dec 16, 2021
Hi @travisrandolph-bestbuy, the updating is normal behavior for config connector and can't be changed; all of our resources attempt to reconcile roughly every 10 minutes.
As for the warnings, can you provide some more detail? Is that the entirety of the warning message?
mbzomowski commentedon Dec 20, 2021
Also, can you check if your resources have label leaser on, you'll need to turn if off to avoid the API calls for releasing the leases.
travisrandolph-bestbuy commentedon Dec 22, 2021
@mbzomowski We are not doing anything to manage conflict. Below is an example of the update in the logs.
toumorokoshi commentedon Dec 28, 2021
Hello! As a quick clarification of what @mbzomowski was saying: although it is true that there will be an API GET every 10 minutes to ensure that the resource is up to date, it is not necessarily true that the update will occur every 10 minutes: only in the case where there was a change that needed to be applied.
So update requests every 10 minutes implies that there is something up with the resource: either it is not updating properly, or perhaps some erroneous diff detection on the part of Config Connector.
@travisrandolph-bestbuy can you send us a redacted version of the resource so we can see if we can reproduce locally? The output you sent looks to be logs from the controller, which can be helpful for diagnostics but doesn't help us spot issues like whether label leasing is on or reproducing locally.
Redacted output from a
kubebctl get -o yamlwould be perfect.travisrandolph-bestbuy commentedon Jan 4, 2022
@toumorokoshi Here's the output.
toumorokoshi commentedon Jan 5, 2022
Thanks for the info! Based on this annotation:
there is no conflict prevention on, and most likely there is some field that is being detected as being modified over and over again.
Let me reach to our current bug rotation to follow-up.
maqiuyujoyce commentedon Jan 14, 2022
Hi @travisrandolph-bestbuy , sorry for the delayed response. I was trying to create a GKE cluster with the configuration you provided (except for
enableBinaryAuthorization: truedue to some issues that might require further digging), but was not able to reproduce this issue. The created cluster was not updated regularly.On the other hand, I noticed that if there is any update to the cluster, there should be a corresponding log for the update operation. Could you check the GKE operation logs associated with the cluster to get more information about what has been updated and whether the changes in the operations match the yaml?
An example query to search for GKE operations in Logs Explorer would be:
travisrandolph-bestbuy commentedon Jan 18, 2022
@maqiuyujoyce Here is the update I'm seeing regularly.
jcanseco commentedon Jan 18, 2022
Hi @travisrandolph-bestbuy, I think this issue is probably being caused by
spec.releaseChannel.channel: stable. Can you try setting the field toSTABLEinstead?Officially, the field only recognizes all-caps values as per the field's description in the docs. I can file a bug to allow for all-lower-case values as well to avoid this issue in the future, but we probably won't be able to prioritize this improvement for a while if the all-caps values do work as intended.
If using
STABLEis not enough to fix the issue, it's possible that the diff is coming from one of the{{REDACTED}}values. In that case, I would try comparing your K8s resource state against the output ofgcloud container clusters describeand look for any differences. This is, of course, not a perfect way to debug the issue since the two object models are different, but they are similar enough that you can sometimes use them to look for unintended diffs.travisrandolph-bestbuy commentedon Jan 21, 2022
@jcanseco Updating the release channel to use upper case resolved some of our clusters. One of the logs looks like it was trying to update the maintenance window. After looking into it we were leaving the Zulu time specification off the end of the start and end times. Once I updated my configs to '2019-09-02T15:00:00Z', it stopped trying to update. After these changes most of our clusters are fine now. The last problem seems to be with 'masterAuthorizedNetworksConfig'. I've added the mismatch in the second code box below. We can't add the boolean, so I'm not sure we can do anything to fix that.
jcanseco commentedon Jan 24, 2022
Hey @travisrandolph-bestbuy, I'm glad to hear that most of your clusters are no longer seeing the issue.
Re:
masterAuthorizedNetworksConfig.enabled: this is actually one case where the "diff against gcloud output" approach doesn't work perfectly since KCC hides theenabledflag from the user (KCC sets it to true as long asmasterAuthorizedNetworksConfigis set, and sets it to false otherwise).Would you be willing to share the
kubectl get -o yamlandgcloud container clusters describeoutputs of the last problematic cluster(s)? We can try to take a look and help check for unintended diffs.travisrandolph-bestbuy commentedon Jan 27, 2022
@jcanseco After pushing both the prior changes to all clusters our resources are no longer updating every 10 minutes. I'm closing this one out!
jcanseco commentedon Jan 27, 2022
@travisrandolph-bestbuy, that's great to hear! Thanks for keeping us updated.