Skip to content

Commit 38c5f47

Browse files
authored
support defaultJobLabels and action-level jobLabels (#2344)
* add baJobLabels for all dataform actions * add jobLabels support for `Property graphs`, `Data preparations` and `Bigquery unit tests` action types * Remove unreachable snake_case handling from restoreJobLabelsAsWritten loadActionConfigsFile checks actions.yaml with verifyObjectMatchesProto before restoreJobLabelsAsWritten runs, and that check rejects snake_case keys such as incremental_table, data_preparation and job_labels. Only the camelCase action types can reach this function, so drop the snake_case pairs and the job_labels fallback. * Address PR #2344 review comments * Rethrow job label validation errors when no session is provided setActionJobLabels previously used session?.compileError(...), which silently dropped validation errors when called without a session. Record the error on the session when one is provided, and rethrow it otherwise. * Address follow-up review comments on job labels - Proto comments and configs.md refer to the YAML key `defaultJobLabels` instead of `default_job_labels`. - The `defaultJobLabels` docs now say that actions can't remove a default key, and that the CLI doesn't apply the table's `labels` to the jobs of actions with job labels. - Error messages say "lowercase ASCII letters", since international characters aren't allowed. - main_action_configs_test checks that each case compiles exactly one action. - Test that unit tests without `jobLabels` inherit `defaultJobLabels`. * Check BigQuery's 64-label limit before running, and apply --job-labels to unit tests - A new resolveJobLabels helper decides a job's labels in one place: the action's job labels, or its table's bigqueryLabels if it has none, overridden by --job-labels. - dataform run stops before running anything if an action, or a unit test with --run-tests, would get more than 64 labels (BigQuery's per-job limit). Disabled actions are skipped. - dataform run --run-tests now applies --job-labels to unit-test jobs too. * Return validated job labels instead of mutating the input, and tidy job label tests - validateJobLabels takes unknown input and returns a new object with string values, instead of converting values inside the caller's object. core/main.ts stores the returned labels. Its error now says values may be strings, numbers, or booleans. - The "must be an object" check lives in one helper shared by validateJobLabels and setActionJobLabels. - Built-in assertion configs are built with AssertionConfig.create() instead of a type assertion. - The label constants are no longer exported, and the collections are read-only. - Tests: one test per case instead of loops in test bodies; check where setActionJobLabels reports errors; drop a test that couldn't fail; give jobLabels propagation its own build test; stop modifying the shared RUN_TEST_GRAPH fixture. * Only set table metadata when there is a description, columns or table labels to apply Job labels give every action an actionDescriptor, so checking only that a descriptor exists made `dataform run` patch table metadata (tables.get and tables.patch) for every table, view and incremental table once defaultJobLabels was set. Only call setMetadata when there's a description, column descriptions or table labels for it to apply. * Document job label rules and how --job-labels interacts with other labels Document the allowed job label keys and values, the 32-label limit, reserved keys, built-in assertion inheritance and quoting of numeric YAML values in the proto comments and the configs reference. Describe in the CLI help and README that --job-labels also applies to unit tests run with --run-tests and overrides job and table labels that use the same key, and add the new exit code for jobs over BigQuery's 64-label limit. Rename the two test() API tests that were described as "test command" tests, since `dataform test` has no --job-labels flag.
1 parent 6f74e8d commit 38c5f47

40 files changed

Lines changed: 1890 additions & 55 deletions

‎cli/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -206,13 +206,13 @@ Compiles and executes the project against BigQuery (or validates it with `--dry-
206206
| `--execution-timeout` | string | — | Wall-clock deadline for the *entire* run (compile + all actions), e.g. `'10m'`, `'2h'`. When it fires, in-flight actions are cancelled and pending actions are skipped. Off by default. |
207207
| `--jit-timeout` | string | — | Per-model just-in-time (JiT) compilation worker timeout, e.g. `'30s'`, `'2m'`. Each action with JiT code gets its own fresh deadline, independent of `--execution-timeout`. If unset, only `--execution-timeout` bounds JiT work. |
208208
| `--job-prefix` | string | — | Adds an additional prefix to BigQuery job IDs, in the form `dataform-${jobPrefix}-`. |
209-
| `--job-labels` | string | — | Comma-separated `key=value` labels to attach to BigQuery jobs, e.g. `'key1=val1,key2=val2'`. |
209+
| `--job-labels` | string | — | Comma-separated `key=value` labels to attach to BigQuery jobs, e.g. `'key1=val1,key2=val2'`. Also applied to unit tests with `--run-tests`. Overrides `defaultJobLabels`, action `jobLabels` and table `labels` that use the same key. |
210210
| `--emit-lineage` | boolean | — | Emits OpenLineage `RunEvent`s to Knowledge Catalog Lineage for each executed action. Overrides the `lineage.enabled` setting in `workflow_settings.yaml` when specified. See the [Data Lineage guide](../docs/lineage.md). |
211211
| Project-config overrides | — | — | See [Shared project-config flags](#shared-project-config-flags). |
212212

213213
If `--timeout` is set without `--execution-timeout`, `run` prints a warning reminding you that `--timeout` only bounds compilation.
214214

215-
Exit codes: `0` on success, `1` if compilation fails, unit tests fail (with `--run-tests`), the run fails, times out, or is cancelled (Ctrl+C).
215+
Exit codes: `0` on success, `1` if compilation fails, a BigQuery job would get more than 64 labels (checked before any action or unit test runs), unit tests fail (with `--run-tests`), the run fails, times out, or is cancelled (Ctrl+C).
216216

217217
---
218218

‎cli/api/commands/run.ts‎

Lines changed: 71 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ import { LineageEmitter } from "df/cli/api/lineage/emitter";
99
import { Flags } from "df/common/flags";
1010
import { retry } from "df/common/promises";
1111
import { deepClone, equals } from "df/common/protos";
12-
import { targetStringifier } from "df/core/targets";
12+
import { targetAsReadableString, targetStringifier } from "df/core/targets";
1313
import { version } from "df/core/version";
1414
import { dataform } from "df/protos/ts";
1515

@@ -50,6 +50,74 @@ export interface IExecutionOptions {
5050

5151
export type CancelReason = "timeout" | "user" | "cancellation";
5252

53+
/** The maximum number of labels that BigQuery allows on a job. */
54+
export const MAX_BIGQUERY_JOB_LABELS = 64;
55+
56+
/**
57+
* Returns the labels to set on the BigQuery jobs of an action or unit test: its job labels, or its
58+
* table's `bigqueryLabels` if it has no job labels, overridden by `executionLabels` (for example,
59+
* `--job-labels`).
60+
*/
61+
export function resolveJobLabels(
62+
actionDescriptor: dataform.IActionDescriptor | null | undefined,
63+
executionLabels?: { [label: string]: string } | null,
64+
): { [label: string]: string } {
65+
const jobLabels = actionDescriptor?.jobLabels;
66+
// Decoded compiled graphs set unset maps to {}, so check for emptiness rather than presence.
67+
const actionLabels =
68+
jobLabels && Object.keys(jobLabels).length > 0 ? jobLabels : actionDescriptor?.bigqueryLabels;
69+
return { ...(actionLabels || {}), ...(executionLabels || {}) };
70+
}
71+
72+
/**
73+
* Returns an error message if any of the given actions or unit tests would run BigQuery jobs with
74+
* more labels than BigQuery allows, or undefined if none would.
75+
*/
76+
export function checkJobLabelLimit(
77+
actions: dataform.IExecutionAction[],
78+
tests: dataform.ITest[],
79+
executionLabels?: { [label: string]: string } | null,
80+
): string | undefined {
81+
const labelledItems = [
82+
// Like the runner, treat actions with no tasks and no JiT code as disabled: they run no jobs.
83+
...actions
84+
.filter((action) => action.tasks?.length || action.jitCode)
85+
.map((action) => ({
86+
description: `Action "${targetAsReadableString(action.target)}"`,
87+
actionDescriptor: action.actionDescriptor,
88+
})),
89+
...tests.map((testCase) => ({
90+
description: `Unit test "${testCase.name}"`,
91+
actionDescriptor: testCase.actionDescriptor,
92+
})),
93+
];
94+
for (const { description, actionDescriptor } of labelledItems) {
95+
const labelCount = Object.keys(resolveJobLabels(actionDescriptor, executionLabels)).length;
96+
if (labelCount > MAX_BIGQUERY_JOB_LABELS) {
97+
return (
98+
`${description} would run BigQuery jobs with ${labelCount} labels, but BigQuery allows at ` +
99+
`most ${MAX_BIGQUERY_JOB_LABELS}.`
100+
);
101+
}
102+
}
103+
return undefined;
104+
}
105+
106+
/**
107+
* Returns whether `setMetadata` has anything to apply to the table: a description, column
108+
* descriptions or table labels. Other `actionDescriptor` fields, such as `jobLabels`, only affect
109+
* BigQuery jobs, so they don't need a table metadata update.
110+
*/
111+
function hasTableMetadata(
112+
actionDescriptor: dataform.IActionDescriptor | null | undefined,
113+
): boolean {
114+
return (
115+
!!actionDescriptor?.description ||
116+
!!actionDescriptor?.columns?.length ||
117+
Object.keys(actionDescriptor?.bigqueryLabels || {}).length > 0
118+
);
119+
}
120+
53121
export function run(
54122
dbadapter: dbadapters.IDbAdapter,
55123
graph: dataform.IExecutionGraph,
@@ -388,10 +456,7 @@ export class Runner {
388456
return {
389457
dryRun: this.executionOptions.bigquery?.dryRun,
390458
jobPrefix: this.executionOptions.bigquery?.jobPrefix,
391-
labels: {
392-
...(this.executionOptions?.bigquery?.labels || {}),
393-
...(action.actionDescriptor?.bigqueryLabels || {}),
394-
},
459+
labels: resolveJobLabels(action.actionDescriptor, this.executionOptions.bigquery?.labels),
395460
actionRetryLimit: this.executionOptions.bigquery?.actionRetryLimit,
396461
location: this.graph.projectConfig?.defaultLocation || undefined,
397462
reservation:
@@ -489,7 +554,7 @@ export class Runner {
489554
}
490555

491556
if (
492-
action.actionDescriptor &&
557+
hasTableMetadata(action.actionDescriptor) &&
493558
// Only set metadata if we expect the action to complete in SUCCESSFUL state
494559
// (i.e. it must still be RUNNING, and not FAILED).
495560
actionResult.status === dataform.ActionResult.ExecutionStatus.RUNNING &&

‎cli/api/commands/test.ts‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,33 @@
1+
import { resolveJobLabels } from "df/cli/api/commands/run";
12
import * as dbadapters from "df/cli/api/dbadapters";
23
import { dataform } from "df/protos/ts";
34

45
export async function test(
56
dbadapter: dbadapters.IDbAdapter,
67
tests: dataform.ITest[],
8+
executionLabels?: { [label: string]: string },
79
): Promise<dataform.ITestResult[]> {
8-
return await Promise.all(tests.map((testCase) => runTest(dbadapter, testCase)));
10+
return await Promise.all(tests.map((testCase) => runTest(dbadapter, testCase, executionLabels)));
911
}
1012

1113
async function runTest(
1214
dbadapter: dbadapters.IDbAdapter,
1315
testCase: dataform.ITest,
16+
executionLabels?: { [label: string]: string },
1417
): Promise<dataform.ITestResult> {
1518
// TODO: Test results are currently limited to 1MB.
1619
// We should paginate test results to remove this limit.
1720
let actualResults;
1821
let expectedResults;
22+
const labels = resolveJobLabels(testCase.actionDescriptor, executionLabels);
23+
const executeOptions = {
24+
byteLimit: 1024 * 1024,
25+
...(Object.keys(labels).length > 0 ? { bigquery: { labels } } : {}),
26+
};
1927
try {
2028
[actualResults, expectedResults] = await Promise.all([
21-
dbadapter.execute(testCase.testQuery, { byteLimit: 1024 * 1024 }),
22-
dbadapter.execute(testCase.expectedOutputQuery, { byteLimit: 1024 * 1024 }),
29+
dbadapter.execute(testCase.testQuery, executeOptions),
30+
dbadapter.execute(testCase.expectedOutputQuery, executeOptions),
2331
]);
2432
} catch (e) {
2533
return {

‎cli/commands/run_command.ts‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ import yargs from "yargs";
22

33
import { build, credentials, run, test } from "df/cli/api";
44
import { compileForRun } from "df/cli/api/commands/compile";
5+
import { checkJobLabelLimit } from "df/cli/api/commands/run";
56
import { BigQueryDbAdapter } from "df/cli/api/dbadapters/bigquery";
67
import { LineageEmitter } from "df/cli/api/lineage/emitter";
78
import { createLineageEmitter as createLineageEmitterFromFactory } from "df/cli/api/lineage/emitter_factory";
@@ -119,6 +120,17 @@ export const runCommand: ICommand<IRunArgs> = {
119120
dbadapter,
120121
);
121122

123+
// BigQuery rejects jobs with more than 64 labels, so check before anything runs.
124+
const jobLabelLimitError = checkJobLabelLimit(
125+
executionGraph.actions,
126+
argv.runTests ? compiledGraph.tests : [],
127+
argv.jobLabels,
128+
);
129+
if (jobLabelLimitError) {
130+
printError(jobLabelLimitError);
131+
return 1;
132+
}
133+
122134
if (
123135
argv.dryRun &&
124136
isJsonOutput &&
@@ -134,7 +146,7 @@ export const runCommand: ICommand<IRunArgs> = {
134146

135147
if (argv.runTests) {
136148
logger.log(`Running ${compiledGraph.tests.length} unit tests...\n`);
137-
const testResults = await test(dbadapter, compiledGraph.tests);
149+
const testResults = await test(dbadapter, compiledGraph.tests, argv.jobLabels);
138150
testResults.forEach((testResult) => printTestResult(testResult));
139151
if (testResults.some((testResult) => !testResult.successful)) {
140152
printError("\nUnit tests did not pass; aborting run.");

‎cli/commands/run_options.ts‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -151,7 +151,10 @@ export const jobPrefixOption: INamedOption<yargs.Options, IRunArgs> = {
151151
export const bigqueryJobLabelsOption: INamedOption<yargs.Options, IRunArgs> = {
152152
name: "job-labels",
153153
option: {
154-
describe: "Comma-separated list of labels to add to BigQuery jobs, e.g. 'key1=val1,key2=val2'.",
154+
describe:
155+
"Comma-separated list of labels to add to BigQuery jobs, e.g. 'key1=val1,key2=val2'. " +
156+
"Also applied to unit tests with --run-tests. " +
157+
"Overrides job labels and table labels that use the same key.",
155158
type: "string",
156159
coerce: (raw: string | null) => {
157160
const labels: { [key: string]: string } = {};

‎core/actions/assertion.ts‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ import {
1010
resolvableAsActionConfigTarget,
1111
resolvableAsTarget,
1212
resolveActionsConfigFilename,
13+
setActionJobLabels,
1314
toResolvable,
1415
validateQueryString,
1516
} from "df/core/utils";
@@ -173,6 +174,7 @@ export class Assertion extends ActionBuilder<dataform.Assertion> {
173174
}
174175
this.proto.actionDescriptor.metadata = config.metadata;
175176
}
177+
setActionJobLabels(this.proto, config.jobLabels, this.session);
176178
return this;
177179
}
178180

@@ -262,7 +264,10 @@ export class Assertion extends ActionBuilder<dataform.Assertion> {
262264
* Sets the description of this assertion.
263265
*/
264266
public description(description: string) {
265-
this.proto.actionDescriptor = { description };
267+
if (!this.proto.actionDescriptor) {
268+
this.proto.actionDescriptor = {};
269+
}
270+
this.proto.actionDescriptor.description = description;
266271
return this;
267272
}
268273

‎core/actions/assertion_config_test.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ suite("sqlx and JS API config options", ({ afterEach }) => {
2929
description: "description",
3030
hermetic: true,
3131
dependOnDependencyAssertions: true,
32+
jobLabels: {"job_key": "job_val"},
3233
metadata: {
3334
overview: "assertion overview",
3435
extraProperties: {
@@ -74,6 +75,9 @@ SELECT 1`,
7475
},
7576
actionDescriptor: {
7677
description: "description",
78+
jobLabels: {
79+
job_key: "job_val",
80+
},
7781
metadata: {
7882
overview: "assertion overview",
7983
extraProperties: {

‎core/actions/assertion_test.ts‎

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,9 @@ actions:
4444
description: description
4545
hermetic: true
4646
dependOnDependencyAssertions: true
47-
reservation: reservation`,
47+
reservation: reservation
48+
jobLabels:
49+
job_key: job_val`,
4850
);
4951
writeDefinitionFile(projectDir, "action.sql", "SELECT 1");
5052
writeDefinitionFile(projectDir, "operation.sqlx", "SELECT 1");
@@ -68,6 +70,9 @@ actions:
6870
actionDescriptor: {
6971
description: "description",
7072
reservation: "reservation",
73+
jobLabels: {
74+
job_key: "job_val",
75+
},
7176
},
7277
disabled: true,
7378
fileName: "definitions/action.sql",
@@ -317,6 +322,38 @@ actions:
317322
});
318323
});
319324

325+
test("JS assert().description() preserves jobLabels and reservation on actionDescriptor", () => {
326+
const projectDir = tmpDirFixture.createNewTmpDir();
327+
writeWorkflowSettingsFile(projectDir, VALID_WORKFLOW_SETTINGS_YAML);
328+
writeDefinitionFile(
329+
projectDir,
330+
"assertion.js",
331+
`
332+
assert("name", {
333+
reservation: "projects/p/locations/US/reservations/r",
334+
jobLabels: {
335+
env: "prod",
336+
},
337+
})
338+
.description("my assertion description")
339+
.query("SELECT 1 WHERE FALSE");
340+
`,
341+
);
342+
343+
const result = runMainInVm(coreExecutionRequestFromPath(projectDir));
344+
345+
expect(result.compile.compiledGraph.graphErrors.compilationErrors).deep.equals([]);
346+
expect(asPlainObject(result.compile.compiledGraph.assertions[0]?.actionDescriptor)).deep.equals(
347+
{
348+
description: "my assertion description",
349+
reservation: "projects/p/locations/US/reservations/r",
350+
jobLabels: {
351+
env: "prod",
352+
},
353+
},
354+
);
355+
});
356+
320357
suite("Lakehouse (4-part) support and validation", () => {
321358
test("fails compilation when an assertion is configured with a lakehouse block", () => {
322359
const projectDir = tmpDirFixture.createNewTmpDir();

‎core/actions/data_preparation.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ import {
1212
nativeRequire,
1313
resolvableAsTarget,
1414
resolveActionsConfigFilename,
15+
setActionJobLabels,
1516
toResolvable,
1617
validateQueryString,
1718
} from "df/core/utils";
@@ -66,6 +67,8 @@ export class DataPreparation extends ActionBuilder<dataform.DataPreparation> {
6667
} else if (extension === "sqlx") {
6768
this.configureSqlx(session, config);
6869
}
70+
71+
setActionJobLabels(this.proto, config.jobLabels, this.session);
6972
}
7073

7174
/**

‎core/actions/data_preparation_test.ts‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -884,5 +884,66 @@ FROM x
884884
]),
885885
);
886886
});
887+
888+
test(`data preparation supports defaultJobLabels and jobLabels in actions.yaml and sqlx`, () => {
889+
const workflowSettingsYaml = `
890+
defaultProject: defaultProject
891+
defaultDataset: defaultDataset
892+
defaultLocation: US
893+
defaultJobLabels:
894+
env: dev
895+
team: dataprep_team
896+
`;
897+
const projectDir = createSimpleDataPreparationProject(workflowSettingsYaml, false);
898+
fs.writeFileSync(
899+
path.join(projectDir, "definitions/actions.yaml"),
900+
`
901+
actions:
902+
- dataPreparation:
903+
filename: data_preparation.dp.yaml
904+
jobLabels:
905+
env: prod
906+
cost_center: finance`,
907+
);
908+
fs.writeFileSync(path.join(projectDir, "definitions/data_preparation.dp.yaml"), "\n");
909+
fs.writeFileSync(
910+
path.join(projectDir, "definitions/sqlx_dataprep.sqlx"),
911+
`
912+
config {
913+
type: "dataPreparation",
914+
name: "dest",
915+
jobLabels: {
916+
env: "staging",
917+
},
918+
}
919+
FROM x
920+
|> SELECT *
921+
`,
922+
);
923+
924+
const result = runMainInVm(coreExecutionRequestFromPath(projectDir));
925+
926+
expect(result.compile.compiledGraph.graphErrors.compilationErrors).deep.equals([]);
927+
const dpByFile = new Map(
928+
result.compile.compiledGraph.dataPreparations.map((dp) => [dp.fileName, dp]),
929+
);
930+
expect(
931+
asPlainObject(dpByFile.get("definitions/data_preparation.dp.yaml")?.actionDescriptor),
932+
).deep.equals({
933+
jobLabels: {
934+
env: "prod",
935+
team: "dataprep_team",
936+
cost_center: "finance",
937+
},
938+
});
939+
expect(
940+
asPlainObject(dpByFile.get("definitions/sqlx_dataprep.sqlx")?.actionDescriptor),
941+
).deep.equals({
942+
jobLabels: {
943+
env: "staging",
944+
team: "dataprep_team",
945+
},
946+
});
947+
});
887948
});
888949
});

0 commit comments

Comments
 (0)
⌂ 换网址