Repository navigation
feat: Sum and average aggregation queries - #1097
Conversation
This commit introduces the sum aggregation with a simple test to ensure it works
This change modifies the encoding so that the right data reaches the grpc layer
Adds helper functions and a super class so that average and sum can share the same properties.
Add sum and average to the transaction tests here to improve test coverage for transactions.
The return type for this function is wrong. The function returns an Average so we should use Average.
Make alias optional since the query still works without providing an alias.
The transaction tests should not pass if the transaction is rolled back like they do currently. We must not catch errors and instead let the test fail.
Ensure that the addAggregation function works correctly with an additional assertion check.
This reverts commit 970c0f4.
This test ensures that all the aggregate queries are actually the same no matter how you build them
Write tests to effectively document the toProto output of the various aggregate fields
Equivalent tests to count are written for sum in system tests and some more tests are written too to meet requirements outlined by team.
| * | ||
| */ | ||
| toProto(): any { | ||
| const aggregation = Object.assign( |
There was a problem hiding this comment.
Could you just
const aggregation = this.property_ ? {property: {name: this.property_}} : {};| return Object.assign( | ||
| {operator: this.operator}, | ||
| this.alias_ ? {alias: this.alias_} : null, | ||
| aggregationObject |
There was a problem hiding this comment.
You can also drop the intermediate object and replace this line by
{[this.operator]: aggregation}| * | ||
| * @param {string} property | ||
| */ | ||
| constructor(property: string) { |
There was a problem hiding this comment.
nit:
constructor(public property_: string) {and remove the fields + the assignment
The test for sum and snapshot reads should look at the database before the data is created and run the tests based on the database in that state
Add a test block for a dataset with overflow values and a dataset with NaN values.
Aggregate field should be exported from the client so that it can be used easily by users.
…into sum-avg # Conflicts: # src/index.ts
Some idiomatic changes to improve the state of the code in the PR.
Values for sum and average should be different from those of count and these tests provide the right values now. # Conflicts: # system-test/datastore.ts
Average aggregations regarding appearances have been added in tests and correct values have been assigned
At this point the datastore is populated with data about ratings so computations should be done on that instead.
An assertion error should be thrown so that the test doesn’t pass if the request is successful.
The query with the limit will include all data points with the lowest appearance values. This is likely desired, but also important to document.
The rollbacks for the transaction tests ensure that if a test fails then the data gets reset to where it was before.
| sharedQueryOpts.readOptions.readTime = { | ||
| seconds: Math.floor(seconds), | ||
| }; | ||
| } |
There was a problem hiding this comment.
This code was taken from
nodejs-datastore/src/request.ts
Lines 290 to 299 in e7c3aaf
There was a problem hiding this comment.
any reason why we can't refactor this now?
There was a problem hiding this comment.
No reason. It is a choice between having two simpler PRs and one PR that includes both changes. Doing so now is fine though.
| query: Query, | ||
| options: RunQueryStreamOptions = {} | ||
| ): SharedQueryOptions { | ||
| // TODO: This needs to be refactored later: |
There was a problem hiding this comment.
is this to be done in this PR or in a future one?
There was a problem hiding this comment.
Same the above comment. We can do it now if preferred.
| sharedQueryOpts.readOptions.readTime = { | ||
| seconds: Math.floor(seconds), | ||
| }; | ||
| } |
There was a problem hiding this comment.
any reason why we can't refactor this now?
| assert.strictEqual(secondEntities!.length, 3); | ||
| }); | ||
|
|
||
| it('should query the datastore with snapshot read', async () => { |
There was a problem hiding this comment.
is this related to SUM/AVG changes? If not, let's move it into a separate PR
| after(async () => { | ||
| await datastore.delete(keys); | ||
| }); | ||
| it('should run a sum aggregation', async () => { |
There was a problem hiding this comment.
Include overflow word here
Introducing getSharedOptionsOnly allows us to use that function in two different places to avoid a repeated block of code.
This test belongs inside another PR because it is not directly related to sum/avg.
A test should be included that looks at multiple aggregations in a single query.
The tests themselves should include the word overflow so that it is clear they are working with an overflow dataset.
| } | ||
|
|
||
| /** | ||
| * Gets a copy of the Count aggregate field. |
There was a problem hiding this comment.
should this be updated to remove "Count" since it applies to all?
There was a problem hiding this comment.
This description needs to be updated.
| * @returns {AggregateField} | ||
| */ | ||
| alias(alias: string): AggregateField { | ||
| alias(alias?: string): AggregateField { |
There was a problem hiding this comment.
can we add some additional testing on Count now that this is optional?
There was a problem hiding this comment.
Yes. Added in commit called "Added tests for when an empty alias is provided".
| private getSharedQueryOptions( | ||
| query: Query, | ||
| options: RunQueryStreamOptions = {} | ||
| private getSharedOptionsOnly( |
There was a problem hiding this comment.
what is the difference between this method and getSharedQueryOptions? The naming is confusing here.
There was a problem hiding this comment.
getSharedQueryOptions also accepts a query and calls getSharedOptionsOnly. getSharedOptionsOnly is also called for fetches in createReadStream where options need to be built without considering a query.
There was a problem hiding this comment.
Thanks for the explanation! Maybe getSharedQueryOptions and getSharedRequestOptions, or getQueryOptions and getRequestOptions would be a little more readable, wdyt?
There was a problem hiding this comment.
getQueryOptions and getRequestOptions would work really well. The good thing about these two functions is that they are both private so we can change the names later too if need be.
The comment for setting the alias should not make any mention of count since it is agnostic to the aggregation type.
No matter how alias is encoded, the data structures should store the aggregations the same way inside an aggregate field so as not to create any confusion.
Tests for when an empty alias is provided should check that each aggregate query still works.
The sleep function should enable us to test snapshot reads for aggregate queries
Two tests should be explored that evaluate what happens when multiple aggregations are used and when too many aggregations are used.
Shared options functions could be given a better name so that they make more sense to the code that is using them.
ruyadorno
left a comment
There was a problem hiding this comment.
A couple of questions and tweaking a comment but otherwise it's looking good to me 👍
| ]; | ||
|
|
||
| before(async () => { | ||
| // This function is used for testing snapshot reads by getting a time and then sleeping before adding data. |
There was a problem hiding this comment.
I believe it would be important to rephrase this comment so that it explains why we need the sleeping:
is it simulating process time from datastore? is it simulating latency?
There was a problem hiding this comment.
Done. I actually added some code in the latest commit to ensure the test won't be flakey. The intention of this was to be sure that a recorded read time occurred prior to the time of saving the data we wish to query against. This way a snapshot read at the recorded time would be sure to be against data prior to a save. This is now accomplished by reading data from the database, recording the read time, sleeping and then saving to ensure the save time on the server is later than the recorded read time. This will make sure the test isn't flakey.
The code must explain why the sleep function is needed in the test because its purpose should be clear.
A sum/average test uses snapshot reads for an aggregate query. The key here is write code that will guarantee the read time occurs well before all the data is saved in order to ensure the test isn’t flakey.
| timeBeforeDataCreation = Date.now(); | ||
| await sleep(1000); | ||
| // Save for a key so that a read time can be accessed for snapshot reads. | ||
| const dummyData = Object.assign(Object.assign({}, keysToSave[0]), { |
emptyData is a more accurate variable name for the datastore save data
kolea2
left a comment
There was a problem hiding this comment.
lgtm, please wait for @ruyadorno's approval as well before merging. Thanks for all the hard work on this!
This PR demonstrates how we are going to leverage the current framework to support sum/average queries for later use.