Skip to content

Ensure TableId includes project ID if not explicitly set - #1374

Merged
davidrabinowitz merged 7 commits into
GoogleCloudDataproc:masterfrom
ddebowczyk92:ol_projectid
Jun 4, 2025
Merged

davidrabinowitz merged 7 commits into
GoogleCloudDataproc:masterfrom
ddebowczyk92:ol_projectid

Conversation

@ddebowczyk92

Copy link
Copy Markdown
Contributor

Solves #1373

Signed-off-by: Dominik Dębowczyk <dominik.debowczyk@getindata.com>
@davidrabinowitz

Copy link
Copy Markdown
Member

/gcbrun

1 similar comment
@davidrabinowitz

Copy link
Copy Markdown
Member

/gcbrun

@davidrabinowitz davidrabinowitz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please run ./mvnw spotless:apply -Pall and push the new commit

Signed-off-by: Dominik Dębowczyk <dominik.debowczyk@getindata.com>
@davidrabinowitz

Copy link
Copy Markdown
Member

/gcbrun

@davidrabinowitz

Copy link
Copy Markdown
Member

/gcbrun

@davidrabinowitz

Copy link
Copy Markdown
Member

@ddebowczyk92 Please verify the change

@ddebowczyk92

Copy link
Copy Markdown
Contributor Author

@davidrabinowitz LGTM. I verified it by running a Spark app with a JAR built from this PR, without specifying the project name in the table name. Below is a part of the generated OpenLineage event:
"inputs":[{"namespace":"bigquery","name":"gcp-open-lineage-testing.iceberg_namespace.nyc_taxi_with_duration"

@davidrabinowitz

Copy link
Copy Markdown
Member

/gcbrun

@davidrabinowitz

Copy link
Copy Markdown
Member

@ddebowczyk92 the integration test fails on the following error in DSv2:

Step #5 - "integration-tests-3.1": 2025-06-03 03:03:47 DEBUG io.openlineage.spark.agent.util.ReflectionUtils - Can't get class com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.connector.common.BigQueryUtil
Step #5 - "integration-tests-3.1": java.lang.ClassNotFoundException: com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.connector.common.BigQueryUtil
Step #5 - "integration-tests-3.1": 	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
Step #5 - "integration-tests-3.1": 	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
Step #5 - "integration-tests-3.1": 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
Step #5 - "integration-tests-3.1": 	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
Step #5 - "integration-tests-3.1": 	at java.lang.Class.forName0(Native Method)
Step #5 - "integration-tests-3.1": 	at java.lang.Class.forName(Class.java:348)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.shaded.org.apache.commons.lang3.ClassUtils.getClass(ClassUtils.java:532)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.shaded.org.apache.commons.lang3.ClassUtils.getClass(ClassUtils.java:578)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.shaded.org.apache.commons.lang3.ClassUtils.getClass(ClassUtils.java:561)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.agent.util.ReflectionUtils.tryExecuteStaticMethodForClassName(ReflectionUtils.java:21)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.agent.lifecycle.plan.BigQueryNodeOutputVisitor.extractDatasetIdentifierFromTableId(BigQueryNodeOutputVisitor.java:74)
Step #5 - "integration-tests-3.1": 	at java.util.Optional.flatMap(Optional.java:241)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.agent.lifecycle.plan.BigQueryNodeOutputVisitor.getBigQueryTableName(BigQueryNodeOutputVisitor.java:85)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.agent.lifecycle.plan.BigQueryNodeOutputVisitor.getFromSaveIntoDataSourceCommand(BigQueryNodeOutputVisitor.java:65)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.agent.lifecycle.plan.BigQueryNodeOutputVisitor.apply(BigQueryNodeOutputVisitor.java:96)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.agent.lifecycle.plan.BigQueryNodeOutputVisitor.apply(BigQueryNodeOutputVisitor.java:38)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.agent.util.PlanUtils$1.lambda$apply$2(PlanUtils.java:76)
Step #5 - "integration-tests-3.1": 	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
Step #5 - "integration-tests-3.1": 	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
Step #5 - "integration-tests-3.1": 	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
Step #5 - "integration-tests-3.1": 	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
Step #5 - "integration-tests-3.1": 	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
Step #5 - "integration-tests-3.1": 	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
Step #5 - "integration-tests-3.1": 	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
Step #5 - "integration-tests-3.1": 	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.agent.util.PlanUtils$1.apply(PlanUtils.java:93)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.agent.util.PlanUtils$1.apply(PlanUtils.java:54)
Step #5 - "integration-tests-3.1": 	at scala.PartialFunction.applyOrElse(PartialFunction.scala:127)
Step #5 - "integration-tests-3.1": 	at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126)
Step #5 - "integration-tests-3.1": 	at scala.runtime.AbstractPartialFunction.applyOrElse(AbstractPartialFunction.scala:30)
Step #5 - "integration-tests-3.1": 	at scala.PartialFunction$AndThen.applyOrElse(PartialFunction.scala:194)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.lambda$visitLogicalPlan$14(OpenLineageRunEventBuilder.java:322)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.agent.util.ScalaConversionUtils$2.apply(ScalaConversionUtils.java:185)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.lambda$buildOutputDatasets$19(OpenLineageRunEventBuilder.java:347)
Step #5 - "integration-tests-3.1": 	at java.util.Optional.map(Optional.java:215)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildOutputDatasets(OpenLineageRunEventBuilder.java:347)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:201)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.end(SparkSQLExecutionContext.java:144)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$sparkSQLExecEnd$2(OpenLineageSparkListener.java:150)
Step #5 - "integration-tests-3.1": 	at io.openlineage.client.circuitBreaker.NoOpCircuitBreaker.run(NoOpCircuitBreaker.java:27)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.agent.OpenLineageSparkListener.sparkSQLExecEnd(OpenLineageSparkListener.java:147)
Step #5 - "integration-tests-3.1": 	at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:122)
Step #5 - "integration-tests-3.1": 	at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:100)
Step #5 - "integration-tests-3.1": 	at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
Step #5 - "integration-tests-3.1": 	at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
Step #5 - "integration-tests-3.1": 	at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
Step #5 - "integration-tests-3.1": 	at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:115)
Step #5 - "integration-tests-3.1": 	at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:99)
Step #5 - "integration-tests-3.1": 	at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)
Step #5 - "integration-tests-3.1": 	at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)
Step #5 - "integration-tests-3.1": 	at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
Step #5 - "integration-tests-3.1": 	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
Step #5 - "integration-tests-3.1": 	at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)
Step #5 - "integration-tests-3.1": 	at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)
Step #5 - "integration-tests-3.1": 	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381)
Step #5 - "integration-tests-3.1": 	at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)

Is there a bug in ReflectionUtils? We no not shade com.google.cloud.bigquery.connector.common.BigQueryUtil

Signed-off-by: Dominik Dębowczyk <dominik.debowczyk@getindata.com>
@ddebowczyk92

Copy link
Copy Markdown
Contributor Author

@davidrabinowitz I believe the issue stems from this change: #1370 — it's the same root cause as the duplicated inputs I mentioned here. Removing the BigQueryRelationProvider class resolves the issue; I tested it locally. The failure occurs because the support for BigQuery implemented directly in openlineage spark integration is meant for dsv1 connector version.

@davidrabinowitz

Copy link
Copy Markdown
Member

/gcbrun

@davidrabinowitz

Copy link
Copy Markdown
Member

Revert of #1370 was successfully tested on my machine, so I incorporated this revert into this PR.

Signed-off-by: Dominik Dębowczyk <dominik.debowczyk@getindata.com>
@davidrabinowitz

Copy link
Copy Markdown
Member

/gcbrun

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

⌂ 换网址