Skip to content

feat(storage): support for soft delete policies and restore - #9520

Merged
BrennaEpp merged 9 commits into
googleapis:mainfrom
BrennaEpp:feat-softdelete
Apr 11, 2024
Merged

BrennaEpp merged 9 commits into
googleapis:mainfrom
BrennaEpp:feat-softdelete

Conversation

@BrennaEpp

@BrennaEpp BrennaEpp commented Mar 7, 2024 •

Copy link
Copy Markdown
Contributor

Note: test currently fails pending investiagtion into setting generation on restore

Fixes #9397

Note: test currently fails pending investiagtion into setting generation on restore
@BrennaEpp
BrennaEpp requested review from a team March 7, 2024 00:16
@product-auto-label product-auto-label Bot added size: l Pull request size is large. api: storage Issues related to the Cloud Storage API. labels Mar 7, 2024

@tritone tritone left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good. One API change suggestion and a few questions/docs notes.

Comment thread storage/bucket.go Outdated

// SoftDeleted returns a new BucketHandle with the option to include
// soft-deleted items in list results.
func (b *BucketHandle) SoftDeleted(include bool) *BucketHandle {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this only applies (I think) to BucketHandle.Objects did we consider making this functionality a field on query instead? I think that would be preferable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that it is preferable. Changed.

Comment thread storage/storage.go Resolved
Comment thread storage/storage.go Resolved
Comment thread storage/storage.go Outdated
// Restore will restore a soft-deleted object to full object status.
// Note that you must specify a generation to use this method.
// copySourceACL indicates whether the restored object should copy the
// access controls of the source object.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this only relevant for buckets with fine-grained access? If UBLA is enabled, is this ignored or does it cause an error?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the target bucket has UBLA enforced, the user will get an error warning that object ACLs will not be restored and the object will not be restored.

Comment thread storage/storage.go Resolved
Comment thread storage/client.go Resolved
Comment thread storage/grpc_client.go Resolved
Comment thread storage/grpc_client.go
// There is no separate API for PATCH in gRPC.
// Make a GET call first to retrieve ObjectAttrs.
attrs, err := c.GetObject(ctx, bucket, object, defaultGen, nil, nil, opts...)
attrs, err := c.GetObject(ctx, &getObjectParams{bucket, object, defaultGen, nil, nil, false}, opts...)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can ACLs be updated on a soft deleted object?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think soft deleted objects can be updated at all.

@BrennaEpp
BrennaEpp requested a review from JesseLovelace March 15, 2024 07:45

@tritone tritone left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM, a few more minor questions. Let's try to remove the idempotency check for restore before merging.

Comment thread storage/storage.go Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note as well that by default, soft deleted objects will not be listed.

Comment thread storage/storage.go Resolved
Comment thread storage/storage.go Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We really should have samples for this feature; people are going to be so confused otherwise.

Comment thread storage/storage.go Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If generation is not specified, do we get an intelligible error from the server based on whatever the Go client sends by default? Might be good to check this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error should be intelligible, let me check...

GRPC
rpc error: code = InvalidArgument desc = You must specify a generation. error details: name = BadRequest field = desc = You must specify a generation.

Ah, HTTP's is no longer intelligible since it's a required param googleapi: Error 400: Invalid argument., invalid

What do you suggest we do here? Set if o.gen < 0 { gen = 0 } to get the intelligible error? Sending generation=0 results in googleapi: Error 400: Restore request must specify a generation., required

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sending zero seems fine since the default of -1 seems to be a placeholder for the most recent generation. Or we could just return an error without sending the request since a required param is missing. @JesseLovelace what are other languages doing for this case, assuming that it is relevant?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think sending zero to get the intelligible error seems best here

@BrennaEpp
BrennaEpp requested a review from tritone April 10, 2024 21:50

@tritone tritone left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

@BrennaEpp
BrennaEpp enabled auto-merge (squash) April 11, 2024 20:13
@BrennaEpp
BrennaEpp merged commit 985deb2 into googleapis:main Apr 11, 2024
@BrennaEpp
BrennaEpp deleted the feat-softdelete branch April 11, 2024 20:34
@bianpengyuan

Copy link
Copy Markdown

Hello @BrennaEpp @tritone, when do we expect this PR to be released? Seems like the linked release issue has been created for a while now.

@BrennaEpp

Copy link
Copy Markdown
Contributor Author

Hello @BrennaEpp @tritone, when do we expect this PR to be released? Seems like the linked release issue has been created for a while now.

Hi @bianpengyuan, this PR is released now in storage v1.41.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the Cloud Storage API. size: l Pull request size is large.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

storage: implement soft delete

4 participants

⌂ 换网址