Skip to content

[BUG] Avro direct read throws ClassCastException (UTF8String cannot be cast to Long) on BigQuery DATETIME columns in Spark #1520

Closed

Description

@jhsenjaliya

Environment

  • Connector: com.google.cloud.spark:spark-3.5-bigquery:0.44.2 (also affects spark-bigquery-connector-common:0.44.2)
  • Apache Spark Version: 3.4.x / 3.5.x (tested on Spark 3.5.1 and 3.5.3)
  • Java Version: 11, 17, 21
  • BigQuery Storage Read API Data Format: AVRO (.option("readDataFormat", "AVRO"))

Description

When reading a BigQuery table containing a DATETIME column using the BigQuery Storage Read API with readDataFormat = "AVRO", evaluating or collecting the resulting DataFrame throws:
java.lang.ClassCastException: class org.apache.spark.unsafe.types.UTF8String cannot be cast to class java.lang.Long
This occurs because Spark 3.4+ maps BigQuery DATETIME columns to Spark SQL TimestampNTZType (via TimestampNTZTypeConverter), which physically requires a 64-bit integer (Long representing microseconds from epoch) inside Spark Catalyst's InternalRow. However, SchemaConverters.convertByBigQueryType() still decodes the Avro string payload as an org.apache.spark.unsafe.types.UTF8String.
Reading the exact same table with .option("readDataFormat", "ARROW") succeeds because ArrowSchemaConverter$TimestampMicroVectorAccessor calls vector.get(index) and returns a primitive long.

Steps to Reproduce

  1. Create a BigQuery table with a DATETIME column:
    CREATE TABLE localcloud_db.test_datetime (
        id INT64,
        event_datetime DATETIME
    );
    INSERT INTO localcloud_db.test_datetime VALUES (1, DATETIME '2024-01-02 03:04:05.123456');
    
df = (
    spark.read.format("bigquery")
    .option("readDataFormat", "AVRO")
    .load("my_project.localcloud_db.test_datetime")
)
df.collect() # Triggers ClassCastException

Stacktrace:

Caused by: java.lang.ClassCastException: class org.apache.spark.unsafe.types.UTF8String cannot be cast to class java.lang.Long
	at org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getLong(GenericInternalRow.scala:42)
	at org.apache.spark.sql.catalyst.CatalystTypeConverters$TimestampNTZConverter$.toScalaImpl(CatalystTypeConverters.scala:337)
	at org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toScala(CatalystTypeConverters.scala:125)
	at org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toScala(CatalystTypeConverters.scala:279)
	at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:3412)

Root Cause

Schema Mapping: In spark-3.5-bigquery:0.44.2, com.google.cloud.spark.bigquery.v2.TimestampNTZTypeConverter was introduced to support TimestampNTZType for DATETIME:

public DataType toSparkType(LegacySQLTypeName bqType) {
    if (supportsBigQueryType(bqType)) {
        return DataTypes.TimestampNTZType;
    }
    ...
}

This correctly configures the Spark SQL schema field as StructField("event_datetime", TimestampNTZType).

Avro Decoding Wire Format: BigQuery Storage Read API defines DATETIME in Avro schema as: {"type": "string", "logicalType": "datetime"} The binary payload contains UTF-8 ISO-8601 strings (e.g. "2024-01-02T03:04:05.123456").

Avro Row Conversion (SchemaConverters.java): In SchemaConverters.convertByBigQueryType(Field field, Object value, StructField structField):

if (field.getType().equals(LegacySQLTypeName.DATETIME)) {
    return UTF8String.fromBytes(((Utf8) value).getBytes());
}

The converter ignores structField.dataType() and unconditionally returns UTF8String. When Catalyst accesses the TimestampNTZType column via row.getLong(ordinal), it casts values[ordinal] to Long, leading directly to ClassCastException.

Suggested Fix
In com.google.cloud.spark.bigquery.SchemaConverters.java, update convertByBigQueryType to check if the target Spark column expects TimestampNTZType. If so, parse the ISO-8601 Avro string to microseconds since epoch:

if (field.getType().equals(LegacySQLTypeName.DATETIME)) {
    if (structField != null && structField.dataType() instanceof org.apache.spark.sql.types.TimestampNTZType) {
        String iso = value.toString();
        java.time.LocalDateTime ldt = java.time.LocalDateTime.parse(iso);
        long epochSeconds = ldt.toEpochSecond(java.time.ZoneOffset.UTC);
        long micros = (epochSeconds * 1_000_000L) + (ldt.getNano() / 1_000L);
        return micros; // java.lang.Long
    }
    return UTF8String.fromBytes(((Utf8) value).getBytes());
}

Activity

  1. added a commit that references this issue on Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ 换网址