Skip to content

feat(datastore): Adding BeginLater transaction option implementation - #8972

Merged
bhshkh merged 11 commits into
googleapis:mainfrom
bhshkh:feature/new-transaction
Apr 23, 2024
Merged

bhshkh merged 11 commits into
googleapis:mainfrom
bhshkh:feature/new-transaction

Conversation

@bhshkh

@bhshkh bhshkh commented Nov 3, 2023 •

Copy link
Copy Markdown
Contributor

What is new_transaction ?
ReadOptions are the options shared by Datastore read requests.
new_transaction is a consistency type in ReadOptions. When new_transaction is used in a read request, a new transaction is begun for the request.

This PR adds following:

  1. BeginLater transaction option implementation
  2. Mutex for state in the transaction struct

Currently, when user calls NewTransaction method, client library makes a BeginTransaction rpc call and obtains transaction id. Reads (RunQuery / RunAggregationQuery / LookupRequest), Commit and Rollback use this transaction id in ReadOptions of rpc request.

Without BeginLater transaction option: the above behavior remains unchanged.

With BeginLater transaction option:

  1. Calling NewTransaction method does not make a BeginTransaction rpc call.
  2. The first read rpc passes new_transaction as consistency type to obtain transaction id.
  3. If there is no read called before commit / rollback, begin transaction is called to obtain transaction id before commit / rollback rpc.
    Since there is one less round trip to the Datastore service in point 1 above, user may see lesser latencies when using BeginLater transaction option.

@bhshkh
bhshkh requested review from a team November 3, 2023 15:50
@product-auto-label product-auto-label Bot added the size: xl Pull request size is extra large. label Nov 3, 2023
@bhshkh
bhshkh requested a review from enocom November 3, 2023 15:50
@product-auto-label product-auto-label Bot added the api: datastore Issues related to the Datastore API. label Nov 3, 2023
@bhshkh
bhshkh enabled auto-merge (squash) November 3, 2023 15:50
@bhshkh
bhshkh disabled auto-merge November 3, 2023 15:50
@bhshkh bhshkh changed the title feat(datastore): Adding BeginLater transaction option to improve performance feat(datastore): Adding BeginLater transaction option Nov 3, 2023
@bhshkh
bhshkh marked this pull request as draft November 7, 2023 21:41
@product-auto-label product-auto-label Bot added the stale: old Pull request is old and needs attention. label Dec 4, 2023
@product-auto-label product-auto-label Bot added stale: extraold Pull request is critically old and needs prioritization. and removed stale: old Pull request is old and needs attention. labels Jan 3, 2024
@bhshkh
bhshkh marked this pull request as ready for review January 26, 2024 18:44
@bhshkh
bhshkh requested a review from a team January 26, 2024 18:44
@bhshkh
bhshkh marked this pull request as draft January 26, 2024 18:44
@product-auto-label product-auto-label Bot added size: l Pull request size is large. and removed size: xl Pull request size is extra large. labels Feb 5, 2024
@bhshkh
bhshkh marked this pull request as ready for review February 5, 2024 23:57
@bhshkh
bhshkh enabled auto-merge (squash) February 5, 2024 23:57
@bhshkh bhshkh changed the title feat(datastore): Adding BeginLater transaction option feat(datastore): Adding BeginLater transaction option implementation Feb 5, 2024

@danieljbruce danieljbruce left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initial review with a few comments and questions

Comment thread datastore/query.go Resolved
Comment thread datastore/transaction.go Resolved
Comment thread datastore/transaction.go Resolved
@bhshkh
bhshkh force-pushed the feature/new-transaction branch from 4e5dda0 to 4943983 Compare March 28, 2024 21:56
@bhshkh
bhshkh requested a review from danieljbruce March 28, 2024 22:02
@bhshkh bhshkh added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Mar 29, 2024
@bhshkh

bhshkh commented Mar 29, 2024

Copy link
Copy Markdown
Contributor Author

Do not merge until release freeze ends in mid April

@danieljbruce

Copy link
Copy Markdown

Does this client support retry transactions? Like before any of these changes, if the user provides an id when creating a transaction then will calling a read function make a grpc read request with that transaction id like it does in Node Datastore?

@danieljbruce danieljbruce left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added one question about retry transactions to make sure there is parity with the Node client.

@bhshkh

bhshkh commented Apr 5, 2024

Copy link
Copy Markdown
Contributor Author

Does this client support retry transactions? Like before any of these changes, if the user provides an id when creating a transaction then will calling a read function make a grpc read request with that transaction id like it does in Node Datastore?

before and after any of these changes, no. Go client did not have retry transaction support

@bhshkh
bhshkh requested a review from danieljbruce April 16, 2024 00:12

@danieljbruce danieljbruce left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall. Just a few nits and questions.

Comment thread datastore/transaction.go
return func() {}
}

func (t *Transaction) startProgress(id []byte) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe change the function name to setToInProgress or something like that?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved in #10032

settings: &transactionSettings{},
},
{
desc: "[RunAggregationQuery, Get, Put, Commit] BeginLater. RunAggregationQuery passes new_transaction",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this test verifies that runAggregationQuery will pass in new transaction options. Just for my own knowledge though, what change in this PR allows newTransaction options to be passed in for the runAggregationQuery call? Does runAggregationQuery make a call to the new parseReadOptions()?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Other tests to consider if necessary:

  1. Make sure readTime is passed along for read only transactions
  2. Make sure prevId is passed along and used for read/write transactions

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bhshkh
bhshkh merged commit 4067f4e into googleapis:main Apr 23, 2024
@bhshkh
bhshkh deleted the feature/new-transaction branch April 23, 2024 20:50
@bhshkh
bhshkh requested a review from gkevinzheng April 23, 2024 20:52
Comment thread datastore/query_test.go
}

func TestValidateReadOptions(t *testing.T) {
eventualInTxnErr := errors.New("datastore: cannot use EventualConsistency query in a transaction")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Since you're using this exact error in testing as you are in the code, would it be worth it to have this variable be defined in the code so you can import it in the test package, similar to errExpiredTransaction?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved in #10063

Comment thread datastore/transaction.go
return nil
}

// Acquire state lock if transaction has not started

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add to the comment for acquireLock more about what it is, what its return value signifies, and how it's used in other parts of the PR? It does a lot more than simply acquiring the state lock.

Also because this function does a lot more than just acquiring the state lock, could you rename the function to reflect that?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved in #10063

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: datastore Issues related to the Datastore API. do not merge Indicates a pull request not ready for merge, due to either quality or timing. size: l Pull request size is large. stale: extraold Pull request is critically old and needs prioritization.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

⌂ 换网址