Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Migrate BigQuery integration tests from KMS-encrypted test key to ADC
  • Loading branch information
mbiernacik committed Oct 14, 2026
commit 1ae69b3131f8e7900b362aef5bae688a23e25047
9 changes: 9 additions & 0 deletions .bazelrc
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,15 @@ build --test_output=errors --action_env="GTEST_COLOR=1"

build --incompatible_py3_is_default=false

# Integration tests authenticate with Application Default Credentials (ADC), so they need the user's
# gcloud configuration. Scoped to --config=integration (used by scripts/run_integration_tests) so
# that unit tests stay hermetic and cacheable.
test:integration --test_env=HOME
test:integration --test_env=APPDATA
test:integration --test_env=GOOGLE_APPLICATION_CREDENTIALS
test:integration --test_env=CLOUDSDK_CONFIG
test:integration --test_env=GOOGLE_CLOUD_QUOTA_PROJECT

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: ensureAdcAvailable honours GCE_METADATA_HOST/GCE_METADATA_IP/METADATA_SERVER_DETECTION, but they aren't forwarded here, so in the sandbox they'd only work through an explicit --test_env. Worth forwarding them too, for people using metadata emulators?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. --config=integration now also forwards GCE_METADATA_HOST, GCE_METADATA_IP and METADATA_SERVER_DETECTION, which google-auth-library reads too. Bazel doesn't pass variables that aren't set on the client, so CI is unaffected.


# This is an optimisation we used to have to speed up open source development. If open source
# development is too slow, consider setting it back up, so that build results can be shared across
# machines, developers, and workspaces.
Expand Down
4 changes: 0 additions & 4 deletions MODULE.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,3 @@ use_repo(
"org_golang_x_net",
"org_golang_x_text",
)

# Custom gcloud SDK extension
gcloud_sdk_ext = use_extension("//tools/gcloud:extensions.bzl", "gcloud_sdk_extension")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description says tools/gcloud/ was removed, but this PR doesn't touch it. tools/gcloud/secrets.bzl, extensions.bzl and repository_rules.bzl are still there, and with this extension usage gone they're dead code. Could you delete them here (or update the description)?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR does delete tools/gcloud/: all five files (BUILD, extensions.bzl, gcloud_function.bzl, repository_rules.bzl and secrets.bzl) are in the diff as deleted. GitHub just collapses deleted files in "Files changed". On the branch head there's no tools/gcloud/ directory, and nothing references it anymore (git grep -E "tools/gcloud|gcloud_secret|gcloud_sdk" finds nothing).

use_repo(gcloud_sdk_ext, "gcloud_sdk")
17 changes: 0 additions & 17 deletions MODULE.bazel.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion cli/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ ts_test_suite(
"//packages/@dataform/cli:json",
"//packages/@dataform/cli:worker_bundle",
"//packages/@dataform/core:package_tar",
"//test_credentials:bigquery.json",
"//test_credentials:bigquery_credentials",
],
# index_test_base runs `npm` directly, and the CLI runs `npm i` when dataformCoreVersion is set.
include_npm = True,
Expand Down
34 changes: 24 additions & 10 deletions cli/api/dbadapters/bigquery.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
import { BigQuery, GetTablesResponse, TableField, TableMetadata } from "@google-cloud/bigquery";
import {
BigQuery,
BigQueryOptions,
GetTablesResponse,
TableField,
TableMetadata,
} from "@google-cloud/bigquery";
import Long from "long";
import { PromisePoolExecutor } from "promise-pool-executor";

Expand Down Expand Up @@ -39,22 +45,30 @@ export interface IBigQueryExecutionOptions {

export type BigQueryClientProvider = (projectId?: string) => BigQuery;

/**
* Builds the BigQuery client options for the given credentials. When `credentials.credentials`
* (a service account key) is empty, the client authenticates with Application Default Credentials.
*/
export function getBigQueryClientOptions(
credentials: dataform.IBigQuery,
projectId?: string,
): BigQueryOptions {
return {
projectId: projectId || credentials.projectId,
scopes: EXTRA_GOOGLE_SCOPES,
location: credentials.location,
credentials: credentials.credentials && JSON.parse(credentials.credentials),
};
}

export function createBigQueryClientProvider(
credentials: dataform.IBigQuery,
): BigQueryClientProvider {
const clients = new Map<string, BigQuery>();
return (projectId?: string) => {
projectId = projectId || credentials.projectId;
if (!clients.has(projectId)) {
clients.set(
projectId,
new BigQuery({
projectId,
scopes: EXTRA_GOOGLE_SCOPES,
location: credentials.location,
credentials: credentials.credentials && JSON.parse(credentials.credentials),
}),
);
clients.set(projectId, new BigQuery(getBigQueryClientOptions(credentials, projectId)));
}
return clients.get(projectId);
};
Expand Down
71 changes: 69 additions & 2 deletions cli/api/dbadapters/bigquery_test.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,79 @@
import { Dataset, Table } from "@google-cloud/bigquery";
import { expect } from "chai";
import { anything, instance, mock, verify, when } from "ts-mockito";
import { anything, instance, mock, when } from "ts-mockito";

import { BigQueryDbAdapter } from "df/cli/api/dbadapters/bigquery";
import {
BigQueryDbAdapter,
createBigQueryClientProvider,
getBigQueryClientOptions,
} from "df/cli/api/dbadapters/bigquery";
import { dataform } from "df/protos/ts";
import { suite, test } from "df/testing";

suite("BigQueryDbAdapter", () => {
suite("getBigQueryClientOptions", () => {
test("passes the parsed service account key to the client", () => {
const serviceAccountKey = {
type: "service_account",
project_id: "key-project-id",
private_key_id: "key-id-123",
private_key:
"-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC\n-----END PRIVATE KEY-----\n",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the PEM-looking private_key may trigger secret scanners (e.g. GitHub push protection or internal scanners). The test only checks pass-through, so an obviously fake value like "fake-private-key" would work just as well.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, it's "fake-private-key" now. I changed the same pattern in tests/api/credentials.spec.ts too.

client_email: "test-sa@key-project-id.iam.gserviceaccount.com",
client_id: "1234567890",
};
const options = getBigQueryClientOptions(
dataform.BigQuery.create({
projectId: "target-project-id",
location: "EU",
credentials: JSON.stringify(serviceAccountKey),
}),
);
expect(options).to.deep.include({
projectId: "target-project-id",
location: "EU",
credentials: serviceAccountKey,
});
});

test("leaves out credentials when no key is given, so ADC is used", () => {
const options = getBigQueryClientOptions(
dataform.BigQuery.create({ projectId: "adc-project-id", location: "US" }),
);
expect(options).to.deep.include({ projectId: "adc-project-id", location: "US" });
expect(Boolean(options.credentials)).to.equal(false);
});

test("uses the requested project instead of the default one", () => {
const options = getBigQueryClientOptions(
dataform.BigQuery.create({ projectId: "default-project", location: "US" }),
"other-project",
);
expect(options.projectId).to.equal("other-project");
});

test("throws on malformed service account key JSON", () => {
expect(() =>
getBigQueryClientOptions({
projectId: "target-project-id",
location: "US",
credentials: "{not-valid-json",
}),
).to.throw(SyntaxError);
});
});

test("createBigQueryClientProvider creates and caches one client per project", () => {
const clientProvider = createBigQueryClientProvider(
dataform.BigQuery.create({ projectId: "default-project", location: "EU" }),
);
const defaultClient = clientProvider();
expect(defaultClient.projectId).to.equal("default-project");
expect(defaultClient.location).to.equal("EU");
expect(clientProvider()).to.equal(defaultClient);
expect(clientProvider("other-project").projectId).to.equal("other-project");
});

test("tables() with schema filters correctly", async () => {
const mockBigQuery = mock<any>();
const mockDataset = mock<Dataset>();
Expand Down
4 changes: 3 additions & 1 deletion cli/index_run_e2e_test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,11 @@ import {
} from "df/cli/index_test_base";
import { version } from "df/core/version";
import { suite, test, writeDefinitionFile } from "df/testing";
import { requireAdc } from "df/testing/credentials";
import { TmpDirFixture } from "df/testing/fixtures";

suite("run e2e", ({ afterEach }) => {
suite("run e2e", ({ afterEach, before }) => {
requireAdc(before);
const tmpDirFixture = new TmpDirFixture(afterEach);

test("golden path with package.json", async () => {
Expand Down
26 changes: 7 additions & 19 deletions cli/index_test_base.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,40 +13,28 @@ import {
npmPath,
writeDefinitionFile,
} from "df/testing";
import { readTestCredentialsConfig, resolveCredentialsPath } from "df/testing/credentials";
import { TmpDirFixture } from "df/testing/fixtures";

const DEFAULT_PROJECT = "dataform-open-source";
const DEFAULT_LOCATION = "US";

export const CREDENTIALS_PATH = path.resolve("test_credentials/bigquery.json");
export const CREDENTIALS_PATH = resolveCredentialsPath();

const logger = new Logger(true);
const initialCredentialsConfig = readTestCredentialsConfig(CREDENTIALS_PATH);

function getCredentialsProjectId(): string {
try {
if (fs.existsSync(CREDENTIALS_PATH)) {
const parsed = JSON.parse(fs.readFileSync(CREDENTIALS_PATH, "utf8"));
if (parsed?.projectId) {
return parsed.projectId;
}
}
} catch (e) {
// Fall back to default
if (initialCredentialsConfig?.projectId) {
return initialCredentialsConfig.projectId;
}
logger.log(`Project name not specified; defaulting to ${DEFAULT_PROJECT}`);
return DEFAULT_PROJECT;
}

function getCredentialsLocation(): string {
try {
if (fs.existsSync(CREDENTIALS_PATH)) {
const parsed = JSON.parse(fs.readFileSync(CREDENTIALS_PATH, "utf8"));
if (parsed?.location) {
return parsed.location;
}
}
} catch (e) {
// Fall back to default
if (initialCredentialsConfig?.location) {
return initialCredentialsConfig.location;
}
logger.log(`Location not specified; defaulting to ${DEFAULT_LOCATION}`);
return DEFAULT_LOCATION;
Expand Down
4 changes: 3 additions & 1 deletion cli/tests/jit/index_jit_advanced_test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,11 @@ import { expect } from "chai";

import { CREDENTIALS_PATH, runCli, setupJitProject } from "df/cli/index_test_base";
import { suite, test, writeDefinitionFile } from "df/testing";
import { requireAdc } from "df/testing/credentials";
import { TmpDirFixture } from "df/testing/fixtures";

suite("JiT support advanced", ({ afterEach }) => {
suite("JiT support advanced", ({ afterEach, before }) => {
requireAdc(before);
const tmpDirFixture = new TmpDirFixture(afterEach);

test("JiT preOps and postOps support", async () => {
Expand Down
4 changes: 3 additions & 1 deletion cli/tests/jit/index_jit_dependency_test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,11 @@ import {
setupJitProject,
} from "df/cli/index_test_base";
import { suite, test, writeDefinitionFile } from "df/testing";
import { requireAdc } from "df/testing/credentials";
import { TmpDirFixture } from "df/testing/fixtures";

suite("JiT support dependencies", ({ afterEach }) => {
suite("JiT support dependencies", ({ afterEach, before }) => {
requireAdc(before);
const tmpDirFixture = new TmpDirFixture(afterEach);

test("JiT transitive dependency pruning", async () => {
Expand Down
4 changes: 3 additions & 1 deletion cli/tests/jit/index_jit_main_test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,11 @@ import {
setupJitProject,
} from "df/cli/index_test_base";
import { suite, test, writeDefinitionFile } from "df/testing";
import { requireAdc } from "df/testing/credentials";
import { TmpDirFixture } from "df/testing/fixtures";

suite("JiT support main", ({ afterEach }) => {
suite("JiT support main", ({ afterEach, before }) => {
requireAdc(before);
const tmpDirFixture = new TmpDirFixture(afterEach);

test("compile command includes jitCode in output", async () => {
Expand Down
4 changes: 3 additions & 1 deletion cli/tests/jit/index_jit_runtime_test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,11 @@ import * as path from "path";

import { CREDENTIALS_PATH, runCli, setupJitProject } from "df/cli/index_test_base";
import { suite, test, writeDefinitionFile } from "df/testing";
import { requireAdc } from "df/testing/credentials";
import { TmpDirFixture } from "df/testing/fixtures";

suite("JiT support runtime", ({ afterEach }) => {
suite("JiT support runtime", ({ afterEach, before }) => {
requireAdc(before);
const tmpDirFixture = new TmpDirFixture(afterEach);

test("JiT require() of local files is rejected (GCP parity)", async () => {
Expand Down
34 changes: 26 additions & 8 deletions contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,30 +50,48 @@ You can run all unit tests across the project by running this script:

### Integration Test

To run the CLI integration test against your own GCP project:
Integration tests run against BigQuery using [Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials). No service account keys are stored in the repository.

1. Prepare a credentials JSON file at `test_credentials/bigquery.json`. Set values as follows:
1. **Authenticate with ADC:**

- `projectId`: your GCP project id
- `credentials`: the entire content of your GCP service account key JSON file as a single string (you can generate it with `jq -Rsa < path/to/key.json`).
- `location`: location to use in your project
```bash
gcloud auth application-default login
```

Alternatively, you can set `GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json` in your environment.

2. **Required IAM roles:**

Your Google account (or service account) needs the following IAM roles on the target GCP project:
- `roles/bigquery.jobUser` (BigQuery Job User)
- `roles/bigquery.dataEditor` (BigQuery Data Editor)

3. **Configure the target GCP project (optional for `dataform-open-source` members):**

Example:
By default, the committed `test_credentials/bigquery.default.json` points to `"projectId": "dataform-open-source"` and `"location": "US"`.
To run integration tests against your own GCP project without modifying tracked files, create a git-ignored `test_credentials/bigquery.json` file, which takes precedence over the default. If you already have a `test_credentials/bigquery.json` from the previous setup, it keeps working:

```json
{
"projectId": "my-gcp-project",
"credentials": "{\"type\":\"service_account\",...}",
"location": "US"
}
```

2. You can run all integration tests by using this script:
*(Optional)* If you still want to test with an explicit service account key inside the Dataform credentials file rather than ADC, you can include the `"credentials"` field containing the JSON key serialized as a string (`jq -Rsa < path/to/key.json`).

4. **Run the integration tests:**

```bash
./scripts/run_integration_tests
```

To run a single integration test target directly, pass `--config=integration` so that your ADC environment is forwarded into the test sandbox:

```bash
bazel test --config=integration //tests/integration:bigquery.spec
```

### Lint

Use the following command to check for any linting and formatting errors
Expand Down
9 changes: 0 additions & 9 deletions scripts/decrypt_secret

This file was deleted.

6 changes: 4 additions & 2 deletions scripts/run_integration_tests
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,10 @@ set -e
# Run eslint.
bazel run //:eslint -- '**/*.ts'

# Run integration tests
# Run integration tests (--config=integration forwards the ADC environment, see .bazelrc).
bazel test ... \
--config=integration \

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does CI authenticate now? Before, the Bazel gcloud_secret rule decrypted the key. Now it depends on the build SA reaching the metadata server from inside the test sandbox. Could you confirm that the CI SA has roles/bigquery.jobUser and roles/bigquery.dataEditor on dataform-open-source, and that the metadata server is reachable with USE_CLOUD_BUILD_NETWORK? A link to a green integration run with this change would be great.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since your review I've added service account impersonation, so in CI it works like this:

  1. Cloud Build's bazel step runs scripts/run_tests_on_cloudbuild, which runs scripts/run_integration_tests (bazel test --config=integration).
  2. The tests get the Cloud Build service account's token from the metadata server. Build step containers can reach it (the first step of cloudbuild-test.yaml already relies on it for gcloud auth print-access-token), and Bazel's sandbox doesn't block the network.
  3. With that token, they impersonate dataform-testing@dataform-open-source.iam.gserviceaccount.com, so all BigQuery jobs run as that account.

Permissions:

  • dataform-testing@ has roles/bigquery.admin on dataform-open-source, which includes roles/bigquery.jobUser and roles/bigquery.dataEditor.
  • The Cloud Build service account has roles/iam.serviceAccountTokenCreator on dataform-testing@, and the IAM Service Account Credentials API is enabled.

USE_CLOUD_BUILD_NETWORK=true predates this PR, and nothing in the repo reads it, so it doesn't affect access to the metadata server.

Green integration run on the current head (e2e55fa): Cloud Build 403983ac. It includes the new SESSION_USER() test, which fails if the jobs don't run as dataform-testing@. For an earlier run I also checked INFORMATION_SCHEMA.JOBS_BY_PROJECT: all 146 jobs from that build ran as dataform-testing@, and none as the Cloud Build service account.

--test_tag_filters=integration \
--build_tests_only \
--test_env=USE_CLOUD_BUILD_NETWORK=true
--test_env=USE_CLOUD_BUILD_NETWORK=true \
"$@"
Loading
⌂ 换网址