Repository navigation
Add fixes for BigQueryDataset direct controller - #13352
Conversation
471a523 to
daee2a5
Compare
maqiuyujoyce
left a comment
There was a problem hiding this comment.
Could you rerun the test after you remove BigQueryDataset in racheting.go to verify re-reconciliation behavior?
daee2a5 to
36c74b1
Compare
…kgcp with real GCP
…dependent fixtures
maqiuyujoyce
left a comment
There was a problem hiding this comment.
Overall looks good to me.
A bit surprised that real logs and mock logs are almost identical.
I'm okay with deferring the PATCH question later.
/lgtm
| @@ -21,23 +21,9 @@ | |||
| < PUT https://bigquery.googleapis.com/bigquery/v2/projects/${projectId}/datasets/${datasetID}?alt=json | |||
| --- | |||
| > PATCH https://bigquery.googleapis.com/bigquery/v2/projects/${projectId}/datasets/${datasetID}?alt=json&prettyPrint=false | |||
There was a problem hiding this comment.
Do you think it's safer not to update the operation from PUT to PATCH? I'm not super opinionated, but leaning towards the safer side.
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: maqiuyujoyce The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
3b1b1d6
Summary
This PR updates the
BigQueryDatasetDirect controller andmockbigqueryto eliminate false-positive diffs during reconciliation, avoid redundantPATCHcalls when already up to date, and align mock and real GCP behavior.Key Changes
1.
BigQueryDatasetDirect Controller (pkg/controller/direct/bigquerydataset/)defaults.go,bigquerydataset_controller.go):ApplyBigQueryDatasetGCPDefaultsto defaultspec.maxTimeTravelHoursto168hours (7 days) when unset and preserve server-populatedaccessentries whenspec.accessis omitted.Adapter.FindandAdapter.Updateto check for differences only on specified/managed fields and skip redundantPATCHcalls whenlen(updateMask.Paths) == 0(updating status directly when already up to date).utils.go):foundDiffDatasetAccessEntryandsortAccessEntriesto comparespec.accessentries deterministically, includingView,Routine, andDatasetreferences (!reflect.DeepEqual(...)).cloneBigQueryDatasetMetadateto preserveaccess.Dataset.TargetTypeswhen cloning actual dataset metadata.bigquerydataset_mappings.go):BigQueryDataset_ToMetadataToUpdateso only fields present inupdatePaths(includingaccess,is_case_insensitive,labels, etc.) are populated in thePATCHrequest.2. MockGCP BigQuery (
mockgcp/mockbigquery/datasets.go)accessentries (OWNER/projectOwners,WRITER/projectWriters,READER/projectReaders),maxTimeTravelHours: 168,location: "US",satisfiesPzs, andtype: "DEFAULT"on dataset creation (InsertDataset).PatchDatasetto merge labels (proto.Merge+ empty-value label deletion), defaultMaxTimeTravelHoursto168if unset, and retain existing fields when omitted from partialPATCHpayloads.3. E2E Fixture Golden Logs (
pkg/test/resourcefixture/testdata/...)_http.log,_http_mock.log,_http.diff,_http_mock.diff,_exported.yaml) forBigQueryDatasetfixtures (basicbigquerydataset,bigquerydatasetaccessblock,bigquerydatasetaccessfull,fullybigquerydataset,containerannotations/projectid, andresourceid/userspecifiedresourceid).