Skip to content

Folder / Project - make folder and org id a resourceRef (external or direct) #349

Closed

Description

@tedelwartowski-bestbuy

This enhancement request is similar to what has been requested in #181 however I would like to propose changing how the reference to a folder or org id is handled in both a Folder and Project resource. Currently, reference to a folder or org id is handled via an annotation which makes it hard to maintain referential relationships between folders and projects. I would like to propose switching the reference of the folder or org id to be a resourceRef (external or direct) such that we can then maintain a referential hierarchy directly with our config connector manifests. Without this, maintaining relationships between folders and projects is manually intensive and impedes on our ability to properly implement automation for the platform.

Activity

  1. spew commented on Dec 17, 2020

    @spew
    Contributor

    Thanks @tedelwartowski-bestbuy for this suggestion. What you described is the direction we want to take the product with eventually de-emphasizing the annotation.

  2. tedelwartowski-bestbuy commented on Dec 17, 2020

    @tedelwartowski-bestbuy
    Author

    @spew - thank you for confirming this is something for consideration on the product roadmap. I believe #181 would still be relevant as an interim solution until what has been proposed here can be implemented.

  3. spew commented on Dec 17, 2020

    @spew
    Contributor

    We are looking into accelerating the timeline on #181 @jcanseco for visibility

  4. travisrandolph-bestbuy commented on Jan 20, 2021

    @travisrandolph-bestbuy

    @spew @jcanseco Has there been any progress on this feature?

  5. jcanseco commented on Jan 20, 2021

    @jcanseco
    Member

    Hi @travisrandolph-bestbuy and @tedelwartowski-bestbuy, yes this is something we're working on actively. There's a design proposal undergoing internal review right now meant to address the particular problem described in this issue. We'll let you know when we have more updates on progress.

  6. jcanseco commented on Mar 25, 2021

    @jcanseco
    Member

    Hi all, we've added support for folderRef and organizationRef references in Project and Folder in 1.43.0 which should give you the ability to reference Folder resources by name.

    Please note, however, that there is a known issue where you will not be able to delete a Folder and its child resources at the same time. Otherwise, the deletion will get stuck. You must either delete the parent Folder first and wait for it to be gone from the K8s API Server or delete the child resources first. See the Warning here for more details. We intend to look into how to fix this issue in the future to remove the need to order your deletions.

    Also note that the folder-id and organization-id annotations for Project and Folder are now deprecated as a result. To migrate Project and Folder resources across folders/orgs, please update the folderRef or organizationRef fields instead. Note that migrations from folder to organization and vice versa are still not supported, however.

    To maintain consistency across our resources, we intend to roll out support for projectRef, folderRef, and organizationRef in place of the project-id, folder-id, and organization-id annotations for our other resources over the next few weeks.

  7. yashsaini77 commented on Mar 25, 2021

    @yashsaini77

    So when we want to create a project in a folder so in the manifest file of the project, how do we reference the folder id if we are trying to work in an automated way?

  8. yashsaini77 commented on Mar 25, 2021

    @yashsaini77

    If we want to create a folder and then create a project under that folder then how do we reference the folder in the resource Project's manifest file here also remember we need the automation? @jcanseco

    apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
    kind: Project
    metadata:
    annotations:
    # Set this field after finding the folder number.
    cnrm.cloud.google.com/folder-id: ""
    cnrm.cloud.google.com/auto-create-network: "false"
    name: kcc-dev
    spec:
    name: kcc-dev
    billingAccountRef:
    # Replace "${BILLING_ACCOUNT_ID?}" with the numeric ID for your billing account
    external: ""

  9. jcanseco commented on Mar 25, 2021

    @jcanseco
    Member

    Here's an example:

    apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
    kind: Project
    metadata:
      name: my-project
    spec:
      name: My Project
      folderRef:
        name: my-folder
      billingAccountRef:
        external: "AAAAA-BBBBB-CCCCC-DDDDD"
    ---
    apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
    kind: Folder
    metadata:
      name: my-folder
    spec:
      displayName: My Folder
      organizationRef:
        external: "123456789"
  10. yashsaini77 commented on Mar 25, 2021

    @yashsaini77

    So both resources will be in the same manifest and the dependency will be done by itself like firstly folder will be created and then the project will be created under that folder. right? @jcanseco

  11. jcanseco commented on Mar 25, 2021

    @jcanseco
    Member

    the dependency will be done by itself like firstly folder will be created and then the project will be created under that folder. right?

    Yep that's correct @yashsaini77.

    The Project and Folder don't need to be in the same manifest file though. They can be in different files which you can kubectl apply separately.

  12. tonybenchsci commented on Mar 26, 2021

    @tonybenchsci

    "Also note that the folder-id and organization-id annotations for Project and Folder are now deprecated as a result. "

    @jcanseco @spew We actually worked out a directory structure using the annotation method. Is there no plan to keep both annotation and resourceRef working? Are there migration instructions? (because it sounds like upgrading to v1.43.0 may result in errors now)

    Basically we have a namespace in our KCC GKE cluster where the namespace is annotated using cnrm.cloud.google.com/organization-id, and then just have all the kind: Project resources in this namespace. Will this be broken if we upgrade to the latest KCC version? Note that we don't apply the annotation directly to Projects, just the namespace that they belong to (though from docs I believe that means the Projects no longer inherit the annotation from the namespace after v1.43.0)

    "To maintain consistency across our resources, we intend to roll out support for projectRef, folderRef, and organizationRef in place of the project-id, folder-id, and organization-id annotations for our other resources over the next few weeks."

    Doesn't that mean that the entire doc section in https://cloud.google.com/config-connector/docs/how-to/organizing-resources/overview will be obsolete in a few weeks?

  13. jcanseco commented on Mar 26, 2021

    @jcanseco
    Member

    Hi @tonybenchsci. Your set-up should continue to work just fine. We made sure to keep backwards compatibility in mind when developing this feature.

    Here are the important points:

    • Creating new resources with an annotation and without a reference will continue to work. We will just default a reference value for you based on the annotation. For example, if you create a folder-scoped resource with a folder-id: 12345 annotation, we will default a spec.folderRef.external: 12345 field for you.
    • Creating new resources without an annotation or reference in a namespace that has an annotation will continue to work. We will just default a reference value for you based on the annotation. This will hold true even for future resources that never supported the annotations to begin with but do support the references.
    • Resources that already exist in your cluster which have an annotation and no reference will be given a reference value based on the annotation. This means there should be no issues for existing resources.

    The only real behavioral change is the following:

    • If you want to migrate a Project/Folder resource across folders/orgs, you must update the folderRef/organizationRef field instead of the folder-id/organization-id annotation since annotations are ignored once a resource has a reference.
    • The above also means that you can choose to remove the annotations from your resources once they have a reference if desired.

    Doesn't that mean that the entire doc section in https://cloud.google.com/config-connector/docs/how-to/organizing-resources/overview will be obsolete in a few weeks?

    Sort of, and we do plan to update it. Most of it still holds true though. The key difference is that the source of truth for which project/folder/organization a resource belongs to is now the resource's projectRef/folderRef/organizationRef rather than the resource's project-id/folder-id/organization-id annotation.

    Before, namespace-level annotations were used to default resource-level annotations. Now, namespace-level annotations will be used to default references.

    Resources which already support resource-level annotations will continue to support them as a legacy feature, and they will be converted to references. Future resources will only support references. This is what I meant when I said that annotations will be deprecated.

    @tonybenchsci I hope that alleviates the concerns somewhat. Please let me know if you have any other questions.

  14. tonybenchsci commented on Mar 26, 2021

    @tonybenchsci

    @jcanseco Thank you for the excellent explanation. We'll be upgrading now with the assurance of backwards compatibility; also acknowledging that migrating these resources in the future requires updating the new source-of-truth resourceRef

  15. 16 remaining items

  16. tonybenchsci commented on Apr 27, 2021

    @tonybenchsci

    Gotcha, could you search for any of the following strings:

    1. `error applying changes to resource`
    
    2. `error ensuring resource`
    
    3. `error with status update call to API server`
    
    4. `error with update call to API server`
    

    My suspicion is that the controller is failing to add the spec.projectRef to the existing BigQueryDataset resources as would be expected.

    All of them are of the type 3:

    {"severity":"error","logger":"controller-runtime.manager.controller.bigquerydataset-controller","msg":"Reconciler error","reconciler group":"bigquery.cnrm.cloud.google.com","reconciler kind":"BigQueryDataset","name":"REDACTED","namespace":"REDACTED","error":"error with status update call to API server: BigQueryDataset.bigquery.cnrm.cloud.google.com \"REDACTED\" is invalid: spec.projectRef: Required value"}

  17. jcanseco commented on Apr 27, 2021

    @jcanseco
    Member

    Thanks @tonybenchsci! That last comment helped us reproduce the issue. We'll work on a fix and update this thread accordingly.

  18. tonybenchsci commented on Apr 28, 2021

    @tonybenchsci

    I'm guessing 1.48.0 doesn't have a fix, right?

  19. jcanseco commented on Apr 28, 2021

    @jcanseco
    Member

    Hi @tonybenchsci, unfortunately no. We're working on a fix at the moment and will put out a patch release if we can get it in this week. We'll update you when that happens.

  20. jcanseco commented on Apr 30, 2021

    @jcanseco
    Member

    Hi @tonybenchsci, we should have a fix for you in 1.49.0. Could you try it out once it's out and let us know if it works for you? It should be out by EOD.

  21. tonybenchsci commented on Apr 30, 2021

    @tonybenchsci

    Great thanks @jcanseco
    I see 1.49.0 on the releases page, but it seems incomplete, and it's not seen in gsutil ls gs://configconnector-operator yet.

  22. caieo commented on Apr 30, 2021

    @caieo
    Contributor

    @tonybenchsci, sorry for the delay in updating our release documentation and our operator. We'll have the operator storage bucket updated in an hour or so!

  23. caieo commented on Apr 30, 2021

    @caieo
    Contributor

    @tonybenchsci, 1.49.0 should now be up in gs://configconnector-operator/, please let us know if the fix works for you.

  24. tonybenchsci commented on Apr 30, 2021

    @tonybenchsci

    @caieo @jcanseco It worked after 1.49.0 thanks.
    We're now stuck in a deletion state because of previous attempts that got unblocked as a result of the fix.

    message: 'Delete call failed: error deleting resource: [{0 Error when reading
          or editing Dataset: googleapi: Error 400: Dataset project-id1:bsmining is still
          in use, resourceInUse  []}]'
        reason: DeleteFailed
        status: "False"
    

    There must be a kubectl command to cancel a delete...
    Nvm, now I can annotate with abandon policy and re-acquire

    EDIT: This is now resolved for us. Thanks again for such a prompt turnaround!

  25. jcanseco commented on May 3, 2021

    @jcanseco
    Member

    Hi @tonybenchsci, great, happy to hear your issue has been resolved!

  26. tonybenchsci commented on May 27, 2021

    @tonybenchsci

    @jcanseco Is this expected? admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: error validating container annotations: cannot add/change container annotation cnrm.cloud.google.com/folder-id as it is no longer supported by the resource; set one of [spec.folderRef, spec.organizationRef] instead.

    I was expecting the folder-id annotation on a "kind: Project" to still be able to override the namespace level org-id annotation. And then have it do the automatic defaulting of folderRef, just like how the organizationRef is defaulted via the org-id annotation.

  27. jcanseco commented on May 27, 2021

    @jcanseco
    Member

    Hi @tonybenchsci, yes that is expected behavior. Unfortunately, certain cases become quite complex and confusing from a UX perspective by continuing to allow mutations of container annotations, so it was decided to go for a "stop the bleeding" approach by making container annotations immutable. Note that container annotations had been mutable for only Project and Folder since they could be migrated across parent resources.

    As a side-note: we do allow removal of container annotations once the resource has a hierarchical reference. This is done to allow users to prevent other people from potentially being confused by seeing both an annotation and a reference on a resource.

  28. travisrandolph-bestbuy commented on Nov 15, 2021

    @travisrandolph-bestbuy

    @jcanseco We are ready to close this issue. We have moved our Folders over to the org/folder refs.

  29. maqiuyujoyce commented on Nov 15, 2021

    @maqiuyujoyce
    Collaborator

    Thanks for confirming, @travisrandolph-bestbuy !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ 换网址