Repository navigation
feat(core): add support for 4-part Lakehouse Catalog targets - #2384
Conversation
| return inlineAssertions; | ||
| } | ||
|
|
||
| protected validateLakehouseConfig( |
There was a problem hiding this comment.
"Fill catalog/namespace from defaults, then build catalog.namespace" is written 4 times: validateLakehouseConfig, configTargetToCompiledGraphTarget, actionConfigToCompiledGraphTarget and resolvableAsTarget. core/actions/index.ts grows from 495 to 635 lines. Move the Lakehouse logic (resolving defaults, building the schema, the three validateLakehouse* checks) into its own module, e.g. core/actions/lakehouse.ts, so the rules live in one place and the 4 copies can't drift.
There was a problem hiding this comment.
Done in 555c3b7, extracted Lakehouse default resolution (resolveLakehouseFields, resolveLakehouseSchema) and the three validation helpers (validateLakehouseConfig, validateLakehouseColumns, validateLakehouseBigQueryOptions) into core/actions/lakehouse.ts, and updated core/utils.ts and the action classes to call the shared helpers.
|
Also pushed two follow-up cleanup commits:
|
rafal-hawrylak
left a comment
There was a problem hiding this comment.
LGTM, just rebase and fix conflicts.
…/actions/lakehouse.ts
…omplete Lakehouse refs
… and type ILegacyAssertionConfig.lakehouse
0052ed8 to
c8a0744
Compare
Summary
Adds
@dataform/coreand protobuf support for 4-part BigQuery Lakehouse Catalog (BigLake Iceberg REST Catalog) table identifiers (project.catalog.namespace.table).Changes
protos/configs.proto,protos/core.proto): AddDefaultLakehouseConfig(catalog,namespace) toWorkflowSettings/ProjectConfigandActionConfig.LakehouseConfigtoActionConfig.Target,TableConfig,IncrementalTableConfig,OperationConfig, andDeclarationConfig.core/workflow_settings.ts,core/session.ts,core/utils.ts): Normalizelakehouse: { catalog, namespace }(withdefaultLakehouseConfigfallback) intotarget.schema = "${catalog}.${namespace}"while keepingdataform.Targetwire-compatible. SkipdatasetSuffixon 4-part targets while preservingprojectSuffixandnamePrefix.core/contextables.ts,core/utils.ts): Support 4-partref("project", "catalog", "namespace", "table"),resolve(...), object syntax{ project, catalog, namespace, name }, and"project.catalog.namespace.table"dependency strings.core/actions/*): Supportlakehouseontable,incremental,operations(withhasOutput: true), anddeclaration. Reject unsupported combinations at compile time (views, standalone and inline assertions,schema/datasetoricebergconflicts,labels,clusterBy,requirePartitionFilter,partitionExpirationDays, andbigqueryPolicyTags).Testing
core/actions/table_lakehouse_test.tsandcore/actions/incremental_table_lakehouse_test.ts.workflow_settings_test.ts,utils_test.ts,operation_test.ts,declaration_test.ts,view_test.ts, andassertion_test.ts../scripts/run_testsand./scripts/lintpass.