Skip to content

feat(core): add support for 4-part Lakehouse Catalog targets - #2384

Merged
SergikL merged 7 commits into
mainfrom
lakehouse-pcnt-support
Oct 20, 2026
Merged

SergikL merged 7 commits into
mainfrom
lakehouse-pcnt-support

Conversation

@SergikL

@SergikL SergikL commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds @dataform/core and protobuf support for 4-part BigQuery Lakehouse Catalog (BigLake Iceberg REST Catalog) table identifiers (project.catalog.namespace.table).

Changes

  • Protobufs (protos/configs.proto, protos/core.proto): Add DefaultLakehouseConfig (catalog, namespace) to WorkflowSettings/ProjectConfig and ActionConfig.LakehouseConfig to ActionConfig.Target, TableConfig, IncrementalTableConfig, OperationConfig, and DeclarationConfig.
  • Target Normalization (core/workflow_settings.ts, core/session.ts, core/utils.ts): Normalize lakehouse: { catalog, namespace } (with defaultLakehouseConfig fallback) into target.schema = "${catalog}.${namespace}" while keeping dataform.Target wire-compatible. Skip datasetSuffix on 4-part targets while preserving projectSuffix and namePrefix.
  • 4-Part References (core/contextables.ts, core/utils.ts): Support 4-part ref("project", "catalog", "namespace", "table"), resolve(...), object syntax { project, catalog, namespace, name }, and "project.catalog.namespace.table" dependency strings.
  • Actions & Validations (core/actions/*): Support lakehouse on table, incremental, operations (with hasOutput: true), and declaration. Reject unsupported combinations at compile time (views, standalone and inline assertions, schema/dataset or iceberg conflicts, labels, clusterBy, requirePartitionFilter, partitionExpirationDays, and bigqueryPolicyTags).

Testing

  • Added core/actions/table_lakehouse_test.ts and core/actions/incremental_table_lakehouse_test.ts.
  • Extended unit tests in workflow_settings_test.ts, utils_test.ts, operation_test.ts, declaration_test.ts, view_test.ts, and assertion_test.ts.
  • ./scripts/run_tests and ./scripts/lint pass.

@SergikL
SergikL marked this pull request as ready for review October 16, 2026 08:11
@SergikL
SergikL requested a review from a team as a code owner October 16, 2026 08:11
@SergikL
SergikL requested review from zaptot and removed request for a team October 16, 2026 08:11
Comment thread core/utils.ts Resolved
Comment thread core/actions/index.ts Outdated
return inlineAssertions;
}

protected validateLakehouseConfig(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Fill catalog/namespace from defaults, then build catalog.namespace" is written 4 times: validateLakehouseConfig, configTargetToCompiledGraphTarget, actionConfigToCompiledGraphTarget and resolvableAsTarget. core/actions/index.ts grows from 495 to 635 lines. Move the Lakehouse logic (resolving defaults, building the schema, the three validateLakehouse* checks) into its own module, e.g. core/actions/lakehouse.ts, so the rules live in one place and the 4 copies can't drift.

@SergikL SergikL Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 555c3b7, extracted Lakehouse default resolution (resolveLakehouseFields, resolveLakehouseSchema) and the three validation helpers (validateLakehouseConfig, validateLakehouseColumns, validateLakehouseBigQueryOptions) into core/actions/lakehouse.ts, and updated core/utils.ts and the action classes to call the shared helpers.

@SergikL

SergikL commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator Author

Also pushed two follow-up cleanup commits:

7a01112:

  • Updated invalidRefInputMessage in core/utils.ts to document the 1–4 positional input form and 'catalog' / 'namespace' object properties.
  • Restricted dotted string splitting in resolvableAsActionConfigTarget back to 1–3 parts (so Lakehouse dependencies use object syntax { project, catalog, namespace, name } per the design doc, and domain-scoped strings like "example.com:proj.d.t" continue to fail with invalidRefInputMessage), and removed the unrelated { database, schema } branch.
  • Added a clarifying comment in Session.finalizeSchema (core/session.ts).

0052ed8:

  • Used != null checks in resolvableAsActionConfigTarget for consistency with resolvableAsTarget.
  • Typed ILegacyAssertionConfig.lakehouse as dataform.ActionConfig.ILakehouseConfig in core/actions/assertion.ts.

@rafal-hawrylak rafal-hawrylak left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just rebase and fix conflicts.

@SergikL
SergikL force-pushed the lakehouse-pcnt-support branch from 0052ed8 to c8a0744 Compare October 20, 2026 09:55
@SergikL
SergikL merged commit 1b04df7 into main Oct 20, 2026
14 checks passed
@SergikL
SergikL deleted the lakehouse-pcnt-support branch October 20, 2026 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

⌂ 换网址