Skip to content

Add fixes for BigQueryDataset direct controller - #13352

Merged
anhdle-sso merged 8 commits into
GoogleCloudPlatform:masterfrom
anhdle-sso:bqdataset_newlogic
Oct 5, 2026
Merged

anhdle-sso merged 8 commits into
GoogleCloudPlatform:masterfrom
anhdle-sso:bqdataset_newlogic

Conversation

@anhdle-sso

@anhdle-sso anhdle-sso commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This PR updates the BigQueryDataset Direct controller and mockbigquery to eliminate false-positive diffs during reconciliation, avoid redundant PATCH calls when already up to date, and align mock and real GCP behavior.

Key Changes

1. BigQueryDataset Direct Controller (pkg/controller/direct/bigquerydataset/)

  • GCP Defaults Alignment (defaults.go, bigquerydataset_controller.go):
    • Added ApplyBigQueryDatasetGCPDefaults to default spec.maxTimeTravelHours to 168 hours (7 days) when unset and preserve server-populated access entries when spec.access is omitted.
    • Updated Adapter.Find and Adapter.Update to check for differences only on specified/managed fields and skip redundant PATCH calls when len(updateMask.Paths) == 0 (updating status directly when already up to date).
  • Access Entry Diffing & Cloning (utils.go):
    • Added foundDiffDatasetAccessEntry and sortAccessEntries to compare spec.access entries deterministically, including View, Routine, and Dataset references (!reflect.DeepEqual(...)).
    • Updated cloneBigQueryDatasetMetadate to preserve access.Dataset.TargetTypes when cloning actual dataset metadata.
  • Selective Update Payload (bigquerydataset_mappings.go):
    • Updated BigQueryDataset_ToMetadataToUpdate so only fields present in updatePaths (including access, is_case_insensitive, labels, etc.) are populated in the PATCH request.

2. MockGCP BigQuery (mockgcp/mockbigquery/datasets.go)

  • Added default access entries (OWNER/projectOwners, WRITER/projectWriters, READER/projectReaders), maxTimeTravelHours: 168, location: "US", satisfiesPzs, and type: "DEFAULT" on dataset creation (InsertDataset).
  • Updated PatchDataset to merge labels (proto.Merge + empty-value label deletion), default MaxTimeTravelHours to 168 if unset, and retain existing fields when omitted from partial PATCH payloads.

3. E2E Fixture Golden Logs (pkg/test/resourcefixture/testdata/...)

  • Regenerated E2E HTTP and export golden files (_http.log, _http_mock.log, _http.diff, _http_mock.diff, _exported.yaml) for BigQueryDataset fixtures (basicbigquerydataset, bigquerydatasetaccessblock, bigquerydatasetaccessfull, fullybigquerydataset, containerannotations/projectid, and resourceid/userspecifiedresourceid).
NONE

@anhdle-sso anhdle-sso changed the title Add a few fixes for BigQueryDataset direct controller Add fixes for BigQueryDataset direct controller Sep 22, 2026

@maqiuyujoyce maqiuyujoyce left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you rerun the test after you remove BigQueryDataset in racheting.go to verify re-reconciliation behavior?

maqiuyujoyce
maqiuyujoyce previously approved these changes Sep 26, 2026

@maqiuyujoyce maqiuyujoyce left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good to me.

A bit surprised that real logs and mock logs are almost identical.

I'm okay with deferring the PATCH question later.

/lgtm

@@ -21,23 +21,9 @@
< PUT https://bigquery.googleapis.com/bigquery/v2/projects/${projectId}/datasets/${datasetID}?alt=json
---
> PATCH https://bigquery.googleapis.com/bigquery/v2/projects/${projectId}/datasets/${datasetID}?alt=json&prettyPrint=false

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it's safer not to update the operation from PUT to PATCH? I'm not super opinionated, but leaning towards the safer side.

@maqiuyujoyce

Copy link
Copy Markdown
Collaborator

/lgtm

@google-oss-prow google-oss-prow Bot added the lgtm label Oct 2, 2026
@google-oss-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: maqiuyujoyce

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@anhdle-sso
anhdle-sso added this pull request to the merge queue Oct 5, 2026
Merged via the queue into GoogleCloudPlatform:master with commit 3b1b1d6 Oct 5, 2026
253 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

⌂ 换网址