Skip to content

Releases

0.46.2

Choose a tag to compare

@jphalip jphalip released this 20 Nov 19:47
  • PR #1548: The spark-4.0-bigquery, spark-4.1-bigquery and spark-4.2-bigquery jars now
    include the relocated commons-io and commons-lang3 classes, which lets reads with an
    arrowCompressionCodec run with these connectors.
  • PR #1542: Removed the remaining query pushdown classes (SparkBigQueryPushdown,
    SupportsQueryPushdown, BigQueryPushdownException, BigQueryPushdownUnsupportedException)
    and the spark-bigquery-pushdown modules, which have not been built since query pushdown
    support was removed in 0.43.0 (PR #1367).
  • PR #1549: A custom AccessTokenProvider set with gcpAccessTokenProvider can now come from
    the application's jars (for example added with --jars) when the connector is on the
    cluster's classpath.
  • PR #1552: On Spark 3.4 and above, DataSource V2 scans report the pre-pruning row count from
    BigQuery table metadata (plus any streaming buffer estimate) as the numRowsBeforePruning and
    rawInputRows driver metrics for tables, table snapshots, and materialized query reads. Both
    metrics are omitted for views, materialized views, external tables, snapshotTimeMillis reads,
    and empty tables.
  • PR #1564: Reads now keep requesting rows from a ReadRows stream that is retried after a
    retryable error, including when the error happens before the requested responses arrive.
  • PR #1567: count() and SQL COUNT(*) on a read with snapshotTimeMillis now count the table's
    rows as of the snapshot time, like the rows the read returns. Based on PR #1307, thanks
    @tom-s-powell !

0.46.1

Choose a tag to compare

@dataproc-robot dataproc-robot released this 05 Nov 04:18
  • CVE-2026-89044, CVE-2026-100664, CVE-2026-100659: Netty was upgraded to version 4.2.19.Final
  • CVE-2026-102554: Guava has been upgraded to version 33.7.2-jre
  • CVE-2026-110879: Jackson was upgraded to version 2.22.4
  • The BigQuery REST client used by BigQueryClient.getRestTable() now uses the same HTTP
    transport as the BigQuery client, so it validates TLS against the JVM default truststore and
    honors the connector's proxy settings.
  • JSON was upgraded to version 20260814
  • OpenLineage was upgraded to version 1.55.0
  • google-api-client was upgraded to version 2.9.2

0.46.0

Choose a tag to compare

@dataproc-robot dataproc-robot released this 08 Oct 21:58
  • spark-4.2-bigquery is generally available! Like Spark 4.2, this connector requires at
    least Java 17 runtime.
  • ⚠️ Starting the next connector version, 0.47.0, the DataSource V1 connectors
    (a.k.a spark-bigquery-with-dependencies) will not be released as part of the regular
    connector release.
  • Issue #1520: Fixed ClassCastException when reading BigQuery DATETIME columns with
    readDataFormat=AVRO on Spark 3.4+.
  • Issue #1525: Fixed count() returning an incorrect value on DataSource
    V2 reads when the optimized empty projection path splits more than
    Integer.MAX_VALUE rows into a single partition. Thanks @AntinPhillips !
  • PR #1522: Reading a table that does not exist with the DataSource V2 connector now fails
    with Table <name> not found instead of an [INTERNAL_ERROR] NullPointerException,
    matching the DataSource V1 behavior.

0.45.0

Choose a tag to compare

@dataproc-robot dataproc-robot released this 22 Aug 00:44
  • spark-4.1-bigquery is generally available!
  • Issue #1501: Preserved detailed row errors from failed direct writes
  • Issue #1503: Corrected the documented billing project option to parentProject.
  • PR #1494: Added Support for Change Data Capture (CDC)
  • PR #1510: Handle Arrow StructVectors with 0 child vectors in ArrowColumnBatchPartitionReaderContext
  • PR #1512: Fixed a schema-resolution error on Spark 3.5 when reading a nested
    STRUCT column with a filter Spark cannot push down (e.g. struct_col IS NOT NULL). The DataSource V2 reader now honors Spark's nested schema pruning and
    pushes matching nested selected_fields to the BigQuery Storage Read API.
  • PR #1512: Kept reader-owned Arrow vectors open between reusable columnar
    batches on Spark 4.
  • BigQuery API has been upgraded to version 2.69.0
  • BigQuery Storage API has been upgraded to version 3.30.0
  • GAX has been upgraded to version 2.82.0
  • gRPC has been upgraded to version 1.83.1
  • Guava has been upgraded to version 33.6.0-jre
  • Netty has been upgraded to version 4.2.17.Final
  • Protocol Buffers has been upgraded to version 4.35.1

0.44.2

Choose a tag to compare

@dataproc-robot dataproc-robot released this 21 May 01:24
  • Issue #1389: Fixed Timestamp Conversion from Julian to Gregorian conversion
  • Issue #1476: Fixed BigQuery MERGE statement ambiguity on overwrite when table columns are named target or source.
  • PR #1479: Fixing NotSerializableException thrown on the executors
  • PR #1488: Added timing and Job ID logging to write path operations

0.44.1

Choose a tag to compare

@dataproc-robot dataproc-robot released this 26 Mar 01:43
  • BigQuery API has been upgraded to version 2.60.0
  • BigQuery Storage API has been upgraded to version 3.22.1
  • GAX has been upgraded to version 2.75.0
  • gRPC has been upgraded to version 1.79.0
  • Guava has been upgraded to version 33.5.0-jre
  • Netty has been upgraded to version 4.2.10.Final
  • Protocol Buffers has been upgraded to version 4.34.0
  • google-api-client has been upgraded to version 2.9.0

0.44.0

Choose a tag to compare

@dataproc-robot dataproc-robot released this 12 Feb 02:56
  • Added new connector, spark-4.1-bigquery aimed to be used in Spark 4.1. Like Spark 4.1, this connector requires at
    least Java 17 runtime. It is currently in preview mode.
  • spark-4.0-bigquery is generally available!
  • PR #1445: Add streaming support for Spark DS v2 indirect write.
  • PR #1452: Improved the performance of the dynamic partition overwrite for RANGE_BUCKET partitioned tables.
  • PR #1434: Support per-catalog configuration for project and location for BigQueryCatalog.

0.43.1

Choose a tag to compare

@dataproc-robot dataproc-robot released this 22 Oct 22:54
  • Issue #1417: Fixed ClassCastException in AWS federated identity
  • PR #1432: Fixing packaging issue with the spark-4.0-bigquery connector.

0.43.0

Choose a tag to compare

@dataproc-robot dataproc-robot released this 18 Oct 03:11
  • Added new connector, spark-4.0-bigquery aimed to be used in Spark 4.0. Like Spark 4.0, this connector requires at
    least Java 17 runtime. It is currently in preview mode.
  • PR #1367: Query Pushdown is no longer supported.
  • PR #1369: Catalog enhancements
  • PR #1374: Ensure TableId includes project ID if not explicitly set
  • PR #1376: materializationDataset is now optional to read from views or queries.
  • PR #1380: Fixed ImpersonatedCredentials serialization
  • PR #1381: Added the option to set custom credentials scopes
  • PR #1411: Added Support for SparkSession#executeCommand
  • BigQuery API has been upgraded to version 2.54.0
  • BigQuery Storage API has been upgraded to version 3.16.1
  • GAX has been upgraded to version 2.68.2
  • gRPC has been upgraded to version 1.74.0

0.42.4

Choose a tag to compare

@dataproc-robot dataproc-robot released this 20 Jun 20:15
  • PR #1380: Fixed ImpersonatedCredentials serialization
  • PR #1381: Added the option to set custom credentials scopes
  • PR #1384: Support reading from BigQuery tables with spaces in their names
⌂ 换网址