Repository navigation
Releases
Releases · GoogleCloudDataproc/spark-bigquery-connector
0.46.2
- PR #1548: The
spark-4.0-bigquery,spark-4.1-bigqueryandspark-4.2-bigqueryjars now
include the relocated commons-io and commons-lang3 classes, which lets reads with an
arrowCompressionCodecrun with these connectors. - PR #1542: Removed the remaining query pushdown classes (
SparkBigQueryPushdown,
SupportsQueryPushdown,BigQueryPushdownException,BigQueryPushdownUnsupportedException)
and thespark-bigquery-pushdownmodules, which have not been built since query pushdown
support was removed in 0.43.0 (PR #1367). - PR #1549: A custom
AccessTokenProviderset withgcpAccessTokenProvidercan now come from
the application's jars (for example added with--jars) when the connector is on the
cluster's classpath. - PR #1552: On Spark 3.4 and above, DataSource V2 scans report the pre-pruning row count from
BigQuery table metadata (plus any streaming buffer estimate) as thenumRowsBeforePruningand
rawInputRowsdriver metrics for tables, table snapshots, and materializedqueryreads. Both
metrics are omitted for views, materialized views, external tables,snapshotTimeMillisreads,
and empty tables. - PR #1564: Reads now keep requesting rows from a ReadRows stream that is retried after a
retryable error, including when the error happens before the requested responses arrive. - PR #1567:
count()and SQLCOUNT(*)on a read withsnapshotTimeMillisnow count the table's
rows as of the snapshot time, like the rows the read returns. Based on PR #1307, thanks
@tom-s-powell !
0.46.1
- CVE-2026-89044, CVE-2026-100664, CVE-2026-100659: Netty was upgraded to version 4.2.19.Final
- CVE-2026-102554: Guava has been upgraded to version 33.7.2-jre
- CVE-2026-110879: Jackson was upgraded to version 2.22.4
- The BigQuery REST client used by
BigQueryClient.getRestTable()now uses the same HTTP
transport as the BigQuery client, so it validates TLS against the JVM default truststore and
honors the connector's proxy settings. - JSON was upgraded to version 20260814
- OpenLineage was upgraded to version 1.55.0
- google-api-client was upgraded to version 2.9.2
0.46.0
spark-4.2-bigqueryis generally available! Like Spark 4.2, this connector requires at
least Java 17 runtime.⚠️ Starting the next connector version, 0.47.0, the DataSource V1 connectors
(a.k.a spark-bigquery-with-dependencies) will not be released as part of the regular
connector release.- Issue #1520: Fixed ClassCastException when reading BigQuery DATETIME columns with
readDataFormat=AVROon Spark 3.4+. - Issue #1525: Fixed
count()returning an incorrect value on DataSource
V2 reads when the optimized empty projection path splits more than
Integer.MAX_VALUErows into a single partition. Thanks @AntinPhillips ! - PR #1522: Reading a table that does not exist with the DataSource V2 connector now fails
withTable <name> not foundinstead of an[INTERNAL_ERROR]NullPointerException,
matching the DataSource V1 behavior.
0.45.0
spark-4.1-bigqueryis generally available!- Issue #1501: Preserved detailed row errors from failed direct writes
- Issue #1503: Corrected the documented billing project option to
parentProject. - PR #1494: Added Support for Change Data Capture (CDC)
- PR #1510: Handle Arrow StructVectors with 0 child vectors in ArrowColumnBatchPartitionReaderContext
- PR #1512: Fixed a schema-resolution error on Spark 3.5 when reading a nested
STRUCT column with a filter Spark cannot push down (e.g.struct_col IS NOT NULL). The DataSource V2 reader now honors Spark's nested schema pruning and
pushes matching nestedselected_fieldsto the BigQuery Storage Read API. - PR #1512: Kept reader-owned Arrow vectors open between reusable columnar
batches on Spark 4. - BigQuery API has been upgraded to version 2.69.0
- BigQuery Storage API has been upgraded to version 3.30.0
- GAX has been upgraded to version 2.82.0
- gRPC has been upgraded to version 1.83.1
- Guava has been upgraded to version 33.6.0-jre
- Netty has been upgraded to version 4.2.17.Final
- Protocol Buffers has been upgraded to version 4.35.1
0.44.2
- Issue #1389: Fixed Timestamp Conversion from Julian to Gregorian conversion
- Issue #1476: Fixed BigQuery MERGE statement ambiguity on overwrite when table columns are named
targetorsource. - PR #1479: Fixing NotSerializableException thrown on the executors
- PR #1488: Added timing and Job ID logging to write path operations
0.44.1
- BigQuery API has been upgraded to version 2.60.0
- BigQuery Storage API has been upgraded to version 3.22.1
- GAX has been upgraded to version 2.75.0
- gRPC has been upgraded to version 1.79.0
- Guava has been upgraded to version 33.5.0-jre
- Netty has been upgraded to version 4.2.10.Final
- Protocol Buffers has been upgraded to version 4.34.0
- google-api-client has been upgraded to version 2.9.0
0.44.0
- Added new connector,
spark-4.1-bigqueryaimed to be used in Spark 4.1. Like Spark 4.1, this connector requires at
least Java 17 runtime. It is currently in preview mode. spark-4.0-bigqueryis generally available!- PR #1445: Add streaming support for Spark DS v2 indirect write.
- PR #1452: Improved the performance of the dynamic partition overwrite for RANGE_BUCKET partitioned tables.
- PR #1434: Support per-catalog configuration for project and location for BigQueryCatalog.
0.43.1
0.43.0
- Added new connector,
spark-4.0-bigqueryaimed to be used in Spark 4.0. Like Spark 4.0, this connector requires at
least Java 17 runtime. It is currently in preview mode. - PR #1367: Query Pushdown is no longer supported.
- PR #1369: Catalog enhancements
- PR #1374: Ensure TableId includes project ID if not explicitly set
- PR #1376:
materializationDatasetis now optional to read from views or queries. - PR #1380: Fixed ImpersonatedCredentials serialization
- PR #1381: Added the option to set custom credentials scopes
- PR #1411: Added Support for SparkSession#executeCommand
- BigQuery API has been upgraded to version 2.54.0
- BigQuery Storage API has been upgraded to version 3.16.1
- GAX has been upgraded to version 2.68.2
- gRPC has been upgraded to version 1.74.0