You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Folder / Project - make folder and org id a resourceRef (external or direct) #349
This enhancement request is similar to what has been requested in #181 however I would like to propose changing how the reference to a folder or org id is handled in both a Folder and Project resource. Currently, reference to a folder or org id is handled via an annotation which makes it hard to maintain referential relationships between folders and projects. I would like to propose switching the reference of the folder or org id to be a resourceRef (external or direct) such that we can then maintain a referential hierarchy directly with our config connector manifests. Without this, maintaining relationships between folders and projects is manually intensive and impedes on our ability to properly implement automation for the platform.
Thanks @tedelwartowski-bestbuy for this suggestion. What you described is the direction we want to take the product with eventually de-emphasizing the annotation.
@spew - thank you for confirming this is something for consideration on the product roadmap. I believe #181 would still be relevant as an interim solution until what has been proposed here can be implemented.
Hi @travisrandolph-bestbuy and @tedelwartowski-bestbuy, yes this is something we're working on actively. There's a design proposal undergoing internal review right now meant to address the particular problem described in this issue. We'll let you know when we have more updates on progress.
Hi all, we've added support for folderRef and organizationRef references in Project and Folder in 1.43.0 which should give you the ability to reference Folder resources by name.
Please note, however, that there is a known issue where you will not be able to delete a Folder and its child resources at the same time. Otherwise, the deletion will get stuck. You must either delete the parent Folder first and wait for it to be gone from the K8s API Server or delete the child resources first. See the Warninghere for more details. We intend to look into how to fix this issue in the future to remove the need to order your deletions.
Also note that the folder-id and organization-id annotations for Project and Folder are now deprecated as a result. To migrate Project and Folder resources across folders/orgs, please update the folderRef or organizationRef fields instead. Note that migrations from folder to organization and vice versa are still not supported, however.
To maintain consistency across our resources, we intend to roll out support for projectRef, folderRef, and organizationRef in place of the project-id, folder-id, and organization-id annotations for our other resources over the next few weeks.
So when we want to create a project in a folder so in the manifest file of the project, how do we reference the folder id if we are trying to work in an automated way?
If we want to create a folder and then create a project under that folder then how do we reference the folder in the resource Project's manifest file here also remember we need the automation? @jcanseco
apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: Project
metadata:
annotations:
# Set this field after finding the folder number.
cnrm.cloud.google.com/folder-id: ""
cnrm.cloud.google.com/auto-create-network: "false"
name: kcc-dev
spec:
name: kcc-dev
billingAccountRef:
# Replace "${BILLING_ACCOUNT_ID?}" with the numeric ID for your billing account
external: ""
So both resources will be in the same manifest and the dependency will be done by itself like firstly folder will be created and then the project will be created under that folder. right? @jcanseco
"Also note that the folder-id and organization-id annotations for Project and Folder are now deprecated as a result. "
@jcanseco@spew We actually worked out a directory structure using the annotation method. Is there no plan to keep both annotation and resourceRef working? Are there migration instructions? (because it sounds like upgrading to v1.43.0 may result in errors now)
Basically we have a namespace in our KCC GKE cluster where the namespace is annotated using cnrm.cloud.google.com/organization-id, and then just have all the kind: Project resources in this namespace. Will this be broken if we upgrade to the latest KCC version? Note that we don't apply the annotation directly to Projects, just the namespace that they belong to (though from docs I believe that means the Projects no longer inherit the annotation from the namespace after v1.43.0)
"To maintain consistency across our resources, we intend to roll out support for projectRef, folderRef, and organizationRef in place of the project-id, folder-id, and organization-id annotations for our other resources over the next few weeks."
Hi @tonybenchsci. Your set-up should continue to work just fine. We made sure to keep backwards compatibility in mind when developing this feature.
Here are the important points:
Creating new resources with an annotation and without a reference will continue to work. We will just default a reference value for you based on the annotation. For example, if you create a folder-scoped resource with a folder-id: 12345 annotation, we will default a spec.folderRef.external: 12345 field for you.
Creating new resources without an annotation or reference in a namespace that has an annotation will continue to work. We will just default a reference value for you based on the annotation. This will hold true even for future resources that never supported the annotations to begin with but do support the references.
Resources that already exist in your cluster which have an annotation and no reference will be given a reference value based on the annotation. This means there should be no issues for existing resources.
The only real behavioral change is the following:
If you want to migrate a Project/Folder resource across folders/orgs, you must update the folderRef/organizationRef field instead of the folder-id/organization-id annotation since annotations are ignored once a resource has a reference.
The above also means that you can choose to remove the annotations from your resources once they have a reference if desired.
Sort of, and we do plan to update it. Most of it still holds true though. The key difference is that the source of truth for which project/folder/organization a resource belongs to is now the resource's projectRef/folderRef/organizationRef rather than the resource's project-id/folder-id/organization-id annotation.
Before, namespace-level annotations were used to default resource-level annotations. Now, namespace-level annotations will be used to default references.
Resources which already support resource-level annotations will continue to support them as a legacy feature, and they will be converted to references. Future resources will only support references. This is what I meant when I said that annotations will be deprecated.
@tonybenchsci I hope that alleviates the concerns somewhat. Please let me know if you have any other questions.
@jcanseco Thank you for the excellent explanation. We'll be upgrading now with the assurance of backwards compatibility; also acknowledging that migrating these resources in the future requires updating the new source-of-truth resourceRef
Gotcha, could you search for any of the following strings:
1. `error applying changes to resource`
2. `error ensuring resource`
3. `error with status update call to API server`
4. `error with update call to API server`
My suspicion is that the controller is failing to add the spec.projectRef to the existing BigQueryDataset resources as would be expected.
All of them are of the type 3:
{"severity":"error","logger":"controller-runtime.manager.controller.bigquerydataset-controller","msg":"Reconciler error","reconciler group":"bigquery.cnrm.cloud.google.com","reconciler kind":"BigQueryDataset","name":"REDACTED","namespace":"REDACTED","error":"error with status update call to API server: BigQueryDataset.bigquery.cnrm.cloud.google.com \"REDACTED\" is invalid: spec.projectRef: Required value"}
Hi @tonybenchsci, unfortunately no. We're working on a fix at the moment and will put out a patch release if we can get it in this week. We'll update you when that happens.
Hi @tonybenchsci, we should have a fix for you in 1.49.0. Could you try it out once it's out and let us know if it works for you? It should be out by EOD.
@tonybenchsci, sorry for the delay in updating our release documentation and our operator. We'll have the operator storage bucket updated in an hour or so!
@caieo@jcanseco It worked after 1.49.0 thanks.
We're now stuck in a deletion state because of previous attempts that got unblocked as a result of the fix.
message: 'Delete call failed: error deleting resource: [{0 Error when reading
or editing Dataset: googleapi: Error 400: Dataset project-id1:bsmining is still
in use, resourceInUse []}]'
reason: DeleteFailed
status: "False"
There must be a kubectl command to cancel a delete...
Nvm, now I can annotate with abandon policy and re-acquire
EDIT: This is now resolved for us. Thanks again for such a prompt turnaround!
@jcanseco Is this expected? admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: error validating container annotations: cannot add/change container annotation cnrm.cloud.google.com/folder-id as it is no longer supported by the resource; set one of [spec.folderRef, spec.organizationRef] instead.
I was expecting the folder-id annotation on a "kind: Project" to still be able to override the namespace level org-id annotation. And then have it do the automatic defaulting of folderRef, just like how the organizationRef is defaulted via the org-id annotation.
Hi @tonybenchsci, yes that is expected behavior. Unfortunately, certain cases become quite complex and confusing from a UX perspective by continuing to allow mutations of container annotations, so it was decided to go for a "stop the bleeding" approach by making container annotations immutable. Note that container annotations had been mutable for only Project and Folder since they could be migrated across parent resources.
As a side-note: we do allow removal of container annotations once the resource has a hierarchical reference. This is done to allow users to prevent other people from potentially being confused by seeing both an annotation and a reference on a resource.
Activity
spew commentedon Dec 17, 2020
Thanks @tedelwartowski-bestbuy for this suggestion. What you described is the direction we want to take the product with eventually de-emphasizing the annotation.
tedelwartowski-bestbuy commentedon Dec 17, 2020
@spew - thank you for confirming this is something for consideration on the product roadmap. I believe #181 would still be relevant as an interim solution until what has been proposed here can be implemented.
spew commentedon Dec 17, 2020
We are looking into accelerating the timeline on #181 @jcanseco for visibility
travisrandolph-bestbuy commentedon Jan 20, 2021
@spew @jcanseco Has there been any progress on this feature?
jcanseco commentedon Jan 20, 2021
Hi @travisrandolph-bestbuy and @tedelwartowski-bestbuy, yes this is something we're working on actively. There's a design proposal undergoing internal review right now meant to address the particular problem described in this issue. We'll let you know when we have more updates on progress.
jcanseco commentedon Mar 25, 2021
Hi all, we've added support for
folderRefandorganizationRefreferences inProjectandFolderin 1.43.0 which should give you the ability to referenceFolderresources by name.Please note, however, that there is a known issue where you will not be able to delete a
Folderand its child resources at the same time. Otherwise, the deletion will get stuck. You must either delete the parentFolderfirst and wait for it to be gone from the K8s API Server or delete the child resources first. See theWarninghere for more details. We intend to look into how to fix this issue in the future to remove the need to order your deletions.Also note that the
folder-idandorganization-idannotations forProjectandFolderare now deprecated as a result. To migrateProjectandFolderresources across folders/orgs, please update thefolderRefororganizationReffields instead. Note that migrations from folder to organization and vice versa are still not supported, however.To maintain consistency across our resources, we intend to roll out support for
projectRef,folderRef, andorganizationRefin place of theproject-id,folder-id, andorganization-idannotations for our other resources over the next few weeks.yashsaini77 commentedon Mar 25, 2021
So when we want to create a project in a folder so in the manifest file of the project, how do we reference the folder id if we are trying to work in an automated way?
yashsaini77 commentedon Mar 25, 2021
If we want to create a folder and then create a project under that folder then how do we reference the folder in the resource Project's manifest file here also remember we need the automation? @jcanseco
apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: Project
metadata:
annotations:
# Set this field after finding the folder number.
cnrm.cloud.google.com/folder-id: ""
cnrm.cloud.google.com/auto-create-network: "false"
name: kcc-dev
spec:
name: kcc-dev
billingAccountRef:
# Replace "${BILLING_ACCOUNT_ID?}" with the numeric ID for your billing account
external: ""
jcanseco commentedon Mar 25, 2021
Here's an example:
yashsaini77 commentedon Mar 25, 2021
So both resources will be in the same manifest and the dependency will be done by itself like firstly folder will be created and then the project will be created under that folder. right? @jcanseco
jcanseco commentedon Mar 25, 2021
Yep that's correct @yashsaini77.
The
ProjectandFolderdon't need to be in the same manifest file though. They can be in different files which you cankubectl applyseparately.tonybenchsci commentedon Mar 26, 2021
"Also note that the folder-id and organization-id annotations for Project and Folder are now deprecated as a result. "
@jcanseco @spew We actually worked out a directory structure using the annotation method. Is there no plan to keep both annotation and resourceRef working? Are there migration instructions? (because it sounds like upgrading to v1.43.0 may result in errors now)
Basically we have a namespace in our KCC GKE cluster where the namespace is annotated using
cnrm.cloud.google.com/organization-id, and then just have all thekind: Projectresources in this namespace. Will this be broken if we upgrade to the latest KCC version? Note that we don't apply the annotation directly to Projects, just the namespace that they belong to (though from docs I believe that means the Projects no longer inherit the annotation from the namespace after v1.43.0)"To maintain consistency across our resources, we intend to roll out support for projectRef, folderRef, and organizationRef in place of the project-id, folder-id, and organization-id annotations for our other resources over the next few weeks."
Doesn't that mean that the entire doc section in https://cloud.google.com/config-connector/docs/how-to/organizing-resources/overview will be obsolete in a few weeks?
jcanseco commentedon Mar 26, 2021
Hi @tonybenchsci. Your set-up should continue to work just fine. We made sure to keep backwards compatibility in mind when developing this feature.
Here are the important points:
folder-id: 12345annotation, we will default aspec.folderRef.external: 12345field for you.The only real behavioral change is the following:
Project/Folderresource across folders/orgs, you must update thefolderRef/organizationReffield instead of thefolder-id/organization-idannotation since annotations are ignored once a resource has a reference.Sort of, and we do plan to update it. Most of it still holds true though. The key difference is that the source of truth for which project/folder/organization a resource belongs to is now the resource's
projectRef/folderRef/organizationRefrather than the resource'sproject-id/folder-id/organization-idannotation.Before, namespace-level annotations were used to default resource-level annotations. Now, namespace-level annotations will be used to default references.
Resources which already support resource-level annotations will continue to support them as a legacy feature, and they will be converted to references. Future resources will only support references. This is what I meant when I said that annotations will be deprecated.
@tonybenchsci I hope that alleviates the concerns somewhat. Please let me know if you have any other questions.
tonybenchsci commentedon Mar 26, 2021
@jcanseco Thank you for the excellent explanation. We'll be upgrading now with the assurance of backwards compatibility; also acknowledging that migrating these resources in the future requires updating the new source-of-truth
resourceRef16 remaining items
tonybenchsci commentedon Apr 27, 2021
All of them are of the type 3:
{"severity":"error","logger":"controller-runtime.manager.controller.bigquerydataset-controller","msg":"Reconciler error","reconciler group":"bigquery.cnrm.cloud.google.com","reconciler kind":"BigQueryDataset","name":"REDACTED","namespace":"REDACTED","error":"error with status update call to API server: BigQueryDataset.bigquery.cnrm.cloud.google.com \"REDACTED\" is invalid: spec.projectRef: Required value"}jcanseco commentedon Apr 27, 2021
Thanks @tonybenchsci! That last comment helped us reproduce the issue. We'll work on a fix and update this thread accordingly.
tonybenchsci commentedon Apr 28, 2021
I'm guessing 1.48.0 doesn't have a fix, right?
jcanseco commentedon Apr 28, 2021
Hi @tonybenchsci, unfortunately no. We're working on a fix at the moment and will put out a patch release if we can get it in this week. We'll update you when that happens.
jcanseco commentedon Apr 30, 2021
Hi @tonybenchsci, we should have a fix for you in 1.49.0. Could you try it out once it's out and let us know if it works for you? It should be out by EOD.
tonybenchsci commentedon Apr 30, 2021
Great thanks @jcanseco
I see 1.49.0 on the releases page, but it seems incomplete, and it's not seen in
gsutil ls gs://configconnector-operatoryet.caieo commentedon Apr 30, 2021
@tonybenchsci, sorry for the delay in updating our release documentation and our operator. We'll have the operator storage bucket updated in an hour or so!
caieo commentedon Apr 30, 2021
@tonybenchsci, 1.49.0 should now be up in
gs://configconnector-operator/, please let us know if the fix works for you.tonybenchsci commentedon Apr 30, 2021
@caieo @jcanseco It worked after 1.49.0 thanks.
We're now stuck in a deletion state because of previous attempts that got unblocked as a result of the fix.
There must be a kubectl command to cancel a delete...
Nvm, now I can annotate with abandon policy and re-acquire
EDIT: This is now resolved for us. Thanks again for such a prompt turnaround!
jcanseco commentedon May 3, 2021
Hi @tonybenchsci, great, happy to hear your issue has been resolved!
tonybenchsci commentedon May 27, 2021
@jcanseco Is this expected?
admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: error validating container annotations: cannot add/change container annotation cnrm.cloud.google.com/folder-id as it is no longer supported by the resource; set one of [spec.folderRef, spec.organizationRef] instead.I was expecting the folder-id annotation on a "kind: Project" to still be able to override the namespace level org-id annotation. And then have it do the automatic defaulting of folderRef, just like how the organizationRef is defaulted via the org-id annotation.
jcanseco commentedon May 27, 2021
Hi @tonybenchsci, yes that is expected behavior. Unfortunately, certain cases become quite complex and confusing from a UX perspective by continuing to allow mutations of container annotations, so it was decided to go for a "stop the bleeding" approach by making container annotations immutable. Note that container annotations had been mutable for only
ProjectandFoldersince they could be migrated across parent resources.As a side-note: we do allow removal of container annotations once the resource has a hierarchical reference. This is done to allow users to prevent other people from potentially being confused by seeing both an annotation and a reference on a resource.
travisrandolph-bestbuy commentedon Nov 15, 2021
@jcanseco We are ready to close this issue. We have moved our Folders over to the org/folder refs.
maqiuyujoyce commentedon Nov 15, 2021
Thanks for confirming, @travisrandolph-bestbuy !