You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 38c5f47
Browse filesBrowse the repository at this point in the historyBrowse files
support defaultJobLabels and action-level jobLabels (#2344)
* add baJobLabels for all dataform actions
* add jobLabels support for `Property graphs`, `Data preparations` and `Bigquery unit tests` action types
* Remove unreachable snake_case handling from restoreJobLabelsAsWritten
loadActionConfigsFile checks actions.yaml with verifyObjectMatchesProto
before restoreJobLabelsAsWritten runs, and that check rejects snake_case
keys such as incremental_table, data_preparation and job_labels. Only the
camelCase action types can reach this function, so drop the snake_case
pairs and the job_labels fallback.
* Address PR #2344 review comments
* Rethrow job label validation errors when no session is provided
setActionJobLabels previously used session?.compileError(...), which silently dropped validation errors when called without a session. Record the error on the session when one is provided, and rethrow it otherwise.
* Address follow-up review comments on job labels
- Proto comments and configs.md refer to the YAML key `defaultJobLabels` instead of `default_job_labels`.
- The `defaultJobLabels` docs now say that actions can't remove a default key, and that the CLI doesn't apply the table's `labels` to the jobs of actions with job labels.
- Error messages say "lowercase ASCII letters", since international characters aren't allowed.
- main_action_configs_test checks that each case compiles exactly one action.
- Test that unit tests without `jobLabels` inherit `defaultJobLabels`.
* Check BigQuery's 64-label limit before running, and apply --job-labels to unit tests
- A new resolveJobLabels helper decides a job's labels in one place: the action's job labels, or its table's bigqueryLabels if it has none, overridden by --job-labels.
- dataform run stops before running anything if an action, or a unit test with --run-tests, would get more than 64 labels (BigQuery's per-job limit). Disabled actions are skipped.
- dataform run --run-tests now applies --job-labels to unit-test jobs too.
* Return validated job labels instead of mutating the input, and tidy job label tests
- validateJobLabels takes unknown input and returns a new object with string values, instead of converting values inside the caller's object. core/main.ts stores the returned labels. Its error now says values may be strings, numbers, or booleans.
- The "must be an object" check lives in one helper shared by validateJobLabels and setActionJobLabels.
- Built-in assertion configs are built with AssertionConfig.create() instead of a type assertion.
- The label constants are no longer exported, and the collections are read-only.
- Tests: one test per case instead of loops in test bodies; check where setActionJobLabels reports errors; drop a test that couldn't fail; give jobLabels propagation its own build test; stop modifying the shared RUN_TEST_GRAPH fixture.
* Only set table metadata when there is a description, columns or table labels to apply
Job labels give every action an actionDescriptor, so checking only that a
descriptor exists made `dataform run` patch table metadata (tables.get and
tables.patch) for every table, view and incremental table once
defaultJobLabels was set. Only call setMetadata when there's a description,
column descriptions or table labels for it to apply.
* Document job label rules and how --job-labels interacts with other labels
Document the allowed job label keys and values, the 32-label limit,
reserved keys, built-in assertion inheritance and quoting of numeric
YAML values in the proto comments and the configs reference. Describe in
the CLI help and README that --job-labels also applies to unit tests run
with --run-tests and overrides job and table labels that use the same key,
and add the new exit code for jobs over BigQuery's 64-label limit.
Rename the two test() API tests that were described as "test command"
tests, since `dataform test` has no --job-labels flag.
Copy file name to clipboardExpand all lines: cli/README.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -206,13 +206,13 @@ Compiles and executes the project against BigQuery (or validates it with `--dry-
206
206
|`--execution-timeout`| string | — | Wall-clock deadline for the *entire* run (compile + all actions), e.g. `'10m'`, `'2h'`. When it fires, in-flight actions are cancelled and pending actions are skipped. Off by default. |
207
207
|`--jit-timeout`| string | — | Per-model just-in-time (JiT) compilation worker timeout, e.g. `'30s'`, `'2m'`. Each action with JiT code gets its own fresh deadline, independent of `--execution-timeout`. If unset, only `--execution-timeout` bounds JiT work. |
208
208
|`--job-prefix`| string | — | Adds an additional prefix to BigQuery job IDs, in the form `dataform-${jobPrefix}-`. |
209
-
|`--job-labels`| string | — | Comma-separated `key=value` labels to attach to BigQuery jobs, e.g. `'key1=val1,key2=val2'`. |
209
+
|`--job-labels`| string | — | Comma-separated `key=value` labels to attach to BigQuery jobs, e.g. `'key1=val1,key2=val2'`. Also applied to unit tests with `--run-tests`. Overrides `defaultJobLabels`, action `jobLabels` and table `labels` that use the same key. |
210
210
|`--emit-lineage`| boolean | — | Emits OpenLineage `RunEvent`s to Knowledge Catalog Lineage for each executed action. Overrides the `lineage.enabled` setting in `workflow_settings.yaml` when specified. See the [Data Lineage guide](../docs/lineage.md). |
If `--timeout` is set without `--execution-timeout`, `run` prints a warning reminding you that `--timeout` only bounds compilation.
214
214
215
-
Exit codes: `0` on success, `1` if compilation fails, unit tests fail (with `--run-tests`), the run fails, times out, or is cancelled (Ctrl+C).
215
+
Exit codes: `0` on success, `1` if compilation fails, a BigQuery job would get more than 64 labels (checked before any action or unit test runs), unit tests fail (with `--run-tests`), the run fails, times out, or is cancelled (Ctrl+C).
0 commit comments