Repository navigation
Description
Environment
- Connector:
com.google.cloud.spark:spark-3.5-bigquery:0.44.2(also affectsspark-bigquery-connector-common:0.44.2) - Apache Spark Version: 3.4.x / 3.5.x (tested on Spark 3.5.1 and 3.5.3)
- Java Version: 11, 17, 21
- BigQuery Storage Read API Data Format:
AVRO(.option("readDataFormat", "AVRO"))
Description
When reading a BigQuery table containing a DATETIME column using the BigQuery Storage Read API with readDataFormat = "AVRO", evaluating or collecting the resulting DataFrame throws:
java.lang.ClassCastException: class org.apache.spark.unsafe.types.UTF8String cannot be cast to class java.lang.Long
This occurs because Spark 3.4+ maps BigQuery DATETIME columns to Spark SQL TimestampNTZType (via TimestampNTZTypeConverter), which physically requires a 64-bit integer (Long representing microseconds from epoch) inside Spark Catalyst's InternalRow. However, SchemaConverters.convertByBigQueryType() still decodes the Avro string payload as an org.apache.spark.unsafe.types.UTF8String.
Reading the exact same table with .option("readDataFormat", "ARROW") succeeds because ArrowSchemaConverter$TimestampMicroVectorAccessor calls vector.get(index) and returns a primitive long.
Steps to Reproduce
- Create a BigQuery table with a
DATETIMEcolumn:CREATE TABLE localcloud_db.test_datetime ( id INT64, event_datetime DATETIME ); INSERT INTO localcloud_db.test_datetime VALUES (1, DATETIME '2024-01-02 03:04:05.123456');
df = (
spark.read.format("bigquery")
.option("readDataFormat", "AVRO")
.load("my_project.localcloud_db.test_datetime")
)
df.collect() # Triggers ClassCastExceptionStacktrace:
Caused by: java.lang.ClassCastException: class org.apache.spark.unsafe.types.UTF8String cannot be cast to class java.lang.Long
at org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getLong(GenericInternalRow.scala:42)
at org.apache.spark.sql.catalyst.CatalystTypeConverters$TimestampNTZConverter$.toScalaImpl(CatalystTypeConverters.scala:337)
at org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toScala(CatalystTypeConverters.scala:125)
at org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toScala(CatalystTypeConverters.scala:279)
at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:3412)
Root Cause
Schema Mapping: In spark-3.5-bigquery:0.44.2, com.google.cloud.spark.bigquery.v2.TimestampNTZTypeConverter was introduced to support TimestampNTZType for DATETIME:
public DataType toSparkType(LegacySQLTypeName bqType) {
if (supportsBigQueryType(bqType)) {
return DataTypes.TimestampNTZType;
}
...
}This correctly configures the Spark SQL schema field as StructField("event_datetime", TimestampNTZType).
Avro Decoding Wire Format: BigQuery Storage Read API defines DATETIME in Avro schema as: {"type": "string", "logicalType": "datetime"} The binary payload contains UTF-8 ISO-8601 strings (e.g. "2024-01-02T03:04:05.123456").
Avro Row Conversion (SchemaConverters.java): In SchemaConverters.convertByBigQueryType(Field field, Object value, StructField structField):
if (field.getType().equals(LegacySQLTypeName.DATETIME)) {
return UTF8String.fromBytes(((Utf8) value).getBytes());
}The converter ignores structField.dataType() and unconditionally returns UTF8String. When Catalyst accesses the TimestampNTZType column via row.getLong(ordinal), it casts values[ordinal] to Long, leading directly to ClassCastException.
Suggested Fix
In com.google.cloud.spark.bigquery.SchemaConverters.java, update convertByBigQueryType to check if the target Spark column expects TimestampNTZType. If so, parse the ISO-8601 Avro string to microseconds since epoch:
if (field.getType().equals(LegacySQLTypeName.DATETIME)) {
if (structField != null && structField.dataType() instanceof org.apache.spark.sql.types.TimestampNTZType) {
String iso = value.toString();
java.time.LocalDateTime ldt = java.time.LocalDateTime.parse(iso);
long epochSeconds = ldt.toEpochSecond(java.time.ZoneOffset.UTC);
long micros = (epochSeconds * 1_000_000L) + (ldt.getNano() / 1_000L);
return micros; // java.lang.Long
}
return UTF8String.fromBytes(((Utf8) value).getBytes());
}
Activity
Issue GoogleCloudDataproc#1520: Fix ClassCastException on Avro direct…