We are happy to present the new 2.78.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.

For more information on changes in 2.78.0, check out the detailed release notes.

Highlights

I/Os

• Added a SchemaTransform for SplunkIO writes, available in Beam YAML as WriteToSplunk (Java/YAML) (#40470).
• (Java) FirestoreIO: added RpcQosOptions.Builder.withRampupThrottlingDisabled(), mirroring the existing DatastoreIO option, to opt out of the 500/50/5 write ramp-up (#27170).
• Added support for attributing BigQuery API quota and billing to a specific GCP project (quota project): quota_project_id parameter in ReadFromBigQuery or --bigquery_quota_project_id pipeline option, honored by the Python BigQuery IO transforms (WriteToBigQuery with FILE_LOADS or STREAMING_INSERTS; not yet STORAGE_WRITE_API, which is Java-backed), --bigQueryQuotaProjectId pipeline option (Java), and bigqueryio.WithQuotaProject read/query option (Go) (#37431).
• (Java/Python/Go) DebeziumIO: readAsJson() can now include op and the remaining Debezium source fields (e.g. MySQL file, pos, row, ts_ms; PostgreSQL lsn, txId, ts_ms) in metadata via withFullMetadata(true) / full_metadata=True / debeziumio.FullMetadata(). Off by default (#40701).
• [IcebergIO] AddFiles is now available through the Managed API with Managed.write(Managed.ICEBERG_ADD_FILES) (Java/Python) (#40827).
• (Java) IcebergIO: AddFiles can register files in several tables in one transform, naming each file's table from its path. Set path_pattern (withPathPattern in Java) to a glob with {name} placeholders, such as gs://lake/export/{table}/**, and make table a template over what it captures, such as default.{table}. The schema pre-pass then runs per table, so a table whose files conflict is refused and its files reported while every other table is still registered. Requires schema_evolution_options and error_handling; not supported with dry_run. Error rows gain a table field only when path_pattern is set, so an existing pipeline's error output is unchanged (#40849).

New Features / Improvements

• (Java) Flink runner: sources now forward Beam metrics to Flink at most once per second instead of after every record, which was a significant share of the CPU of reading. Metrics in the pipeline result are not affected (#40607).
• (Go) Added wait.On, which delays each input window until the corresponding windows in its signal PCollections have closed (#39909).
• (Python) Expanded the SDK worker heap dump (--experiments=enable_heap_dump) with process RSS, CPython allocator/GC stats, and glibc mallinfo2 native-heap/fragmentation stats to help distinguish native-heap from Python-object memory growth (#39244).
• (Python) Added typed decision questions and the EvaluateDecisions transform, with an optional Jev model adapter (#40220).
• The disableCounterMetrics, disableStringSetMetrics and disableBoundedTrieMetrics experiments are now honored by the Python SDK, as they already were in Java (Python) (#38746).
• (Go) Pipeline-level --resource_hints are now forwarded to expansion services, so they apply to cross-language transforms as they do in the Java and Python SDKs. The max_active_bundles_per_worker hint name is now also accepted by --resource_hints (#23893).
• QdrantWriteConfig can now optionally store chunk content in the point payload via content_payload_key, so retrieved points include the chunk text (Python) (#40476).
• ReadFromBigQuery now supports Lakehouse runtime catalog (BigLake metastore) tables with method=DIRECT_READ, using 4-part project.catalog.namespace.table identifiers. Previously project:catalog.namespace.table was silently mis-parsed and tables that report no numBytes failed to split (Python) (#39597).
• (Python) Added AwsOptions with --aws_* pipeline options (e.g. --aws_access_key_id, --aws_region_name, --aws_endpoint_url) that set defaults for all AWS service clients. The existing --s3_* options in S3Options override these defaults for the S3 client only (#40528).
• (Python) Added --aws_credentials_secret_manager so AWS credentials (--aws_access_key_id, --aws_secret_access_key, --aws_session_token and their --s3_* overrides) can be secret specs resolved through a secret manager such as GoogleCloudSecretManager, instead of plain-text values (#40528).
• (Python) Added AwsSecret (AwsSecretsManager / type:awssecret) to retrieve secrets from AWS Secrets Manager, configured from AwsOptions (--aws_*) with optional per-secret connection overrides (#40528).
• (Go) Added schema.RegisterLogicalTypeConversion and schema.RegisterLogicalTypeCoder to register portable logical types in one call, and Go types for the standard beam:logical_type:date:v1 and beam:logical_type:micros_instant:v1 logical types, so schemas from other SDKs with these types can be used in cross-language pipelines (#39684).
• (Python, Java, Go) Added --dataflow_container_base_repository (Python, Go) and --dataflowContainerBaseRepository (Java) to set the repository (e.g. docker.example.com/my-prefix/cloud-dataflow/v1beta3) from which the Dataflow runner pulls its default SDK container images, including the default images of cross-language Apache Beam environments. Defaults to gcr.io/cloud-dataflow/v1beta3 (#40569).
• (Go) GCS and Dataflow clients now support non-default universes such as Trusted Partner Cloud (TPC), using the universe domain from GOOGLE_CLOUD_UNIVERSE_DOMAIN or the default credentials. Added a --gcs_endpoint pipeline option to override the GCS endpoint (#40569).
• (Java) GcsUtil now uses the java-storage client (GcsUtil V2) for GCS operations by default instead of gcsio (GcsUtil V1). To go back to V1, pass --experiments=disable_gcsutil_v2. Pipelines with the use_grpc_for_gcs experiment keep using V1. The use_gcsutil_v2 experiment is no longer needed and is now ignored (#40712).

Breaking Changes

• (Java) Removed Beam SQL Google Cloud Data Catalog table provider (beam-sdks-java-extensions-sql-datacatalog) because GCP Data Catalog has been decommissioned (#40663).
• (Go) The row coder now encodes int16 and uint16 struct fields as 2 byte big endian INT16 values, matching the Java and Python SDKs. This is an update incompatible change for streaming pipelines that use rows with int16 or uint16 fields (#40151).
• (Java) JmsIO migrated to jakarta.jms (JMS 3.1) and ActiveMQ 6.2.5. User code implementing JmsIO.MessageMapper,
  valueMapper, topicNameMapper, or providing a ConnectionFactory must update imports from javax.jms.* to
  jakarta.jms.*. The module now requires Java 17 at runtime (#38729).
• (Java) Dataflow runner: --apiRootUrl no longer defaults to https://dataflow.googleapis.com/. When neither --apiRootUrl nor an absolute --dataflowEndpoint is set, the Google API client library resolves the Dataflow endpoint and honors GOOGLE_CLOUD_UNIVERSE_DOMAIN, so Dataflow jobs can be submitted in non-default universes. Code that reads DataflowPipelineDebugOptions.getApiRootUrl() should handle a null value (#40569).
• (Python) Regex.find_all with group=Regex.ALL now returns every group, as (match, group1, group2, ...), instead of only group 1, and with outputEmpty=False it drops a match only when the whole match is empty, matching the group=0 behavior (#40399).
• (Python) Regex.all_matches now returns every group of the regex, instead of failing with a TypeError when the regex has no groups and dropping nested or unmatched trailing groups (#40562).
• (Python) --dataflow_endpoint now defaults to '' instead of https://dataflow.googleapis.com. When unset, the Google Cloud client library resolves the Dataflow endpoint and honors GOOGLE_CLOUD_UNIVERSE_DOMAIN, so Dataflow jobs can be submitted in non-default universes. Code that reads GoogleCloudOptions.dataflow_endpoint should handle an empty value (#40569).
• (Python) The xgboost extras now install xgboost>=2.0.0,<4 and datatable>=1.1.0,<2, moving both the lower and upper bounds for both. The previous minimum xgboost and datatable versions (1.6.0 and 1.0.0 respectively) are no longer allowed (#40694).
• (Java) [IcebergIO] AddFiles (IcebergAddFiles in YAML) now carries the destination table alongside each file through the registration chain, so that one...

We are happy to present the new 2.77.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.

For more information on changes in 2.77.0, check out the detailed release notes.

Highlights

I/Os

• Added schema_update_options to WriteToBigQuery file loads, allowing BigQuery load jobs to add nullable fields or relax required fields when appending data (Python) (#21141).
• BigQueryIO now supports reading BigQuery Lakehouse runtime catalog (BigLake metastore) Iceberg tables with the Storage Read API, using 4-part project.catalog.namespace.table identifiers (or a TableReference with a composite catalog.namespace dataset id). Previously such references were silently mis-parsed (Java) (#39597) .
• SolaceIO now supports reading and writing binary and text content data payload (Java) (#39875).
• ClickHouseIO: support writing Decimal(P, S) / Decimal32/64/128/256 columns (Java) (#39840).
• SolaceIO now supports reading and writing user properties (message metadata) (Java) (#40099).
• [IcebergIO] AddFiles (IcebergAddFiles in YAML) can evolve the table schema before registering files, with schema_evolution_options, required_columns, incompatible_schema_handling and unverifiable_file_handling (Java/YAML, batch only) (#40144).

New Features / Improvements

• (Java/Python) Watch can bound its deduplication state by event time, retiring an output key once the greatest emitted timestamp has moved more than the allowed lateness past it. Java adds Watch.growthOf(...).withTimestampCursor(). Python adds allowed_lateness for the existing timestamp_cursor option (#18459).
• (Java) Spark Structured Streaming runner: stateful ParDo with state, timers, @RequiresTimeSortedInput and tagged outputs is now supported in batch mode (#39779).
• (Python) Added support for Vertex AI Model Monitoring V2 in RunInference (#39738).
• [Flink Runner] Added opt-in static round-robin split assignment for small bounded sources via the new sourceStaticSplitThresholdMb pipeline option. The default of 0 keeps the existing lazy pull-based assignment (#39873).
• Added automatic caching of bounded, single-pane side-input views for classic Java Flink DataStream execution (#39866).
• (Python) Added Sample.Any, the Python equivalent of Java's Sample.any, which returns up to n arbitrary elements from a PCollection (#18552).

Breaking Changes

• Portable Java SDK now encodes SchemaCoders in a portable way (#34672).
  • Original custom Java coder encoding can still be obtained using StreamingOptions.setUpdateCompatibilityVersion("2.76") (#34672).
  • Fixes (#36496), (#30276), (#29245).
• (Python) TensorRTEngineHandlerNumPy now requires TensorRT 10 or later. TensorRT 8.x is no longer supported, since TensorRT 10 removed the engine binding API the handler was written against (#36306).
  • Engines serialized by TensorRT 8.x must be rebuilt, as an engine can only be deserialized by the major version that built it.
  • TensorRT 10 and later require a GPU with compute capability 7.5 or higher, which excludes NVIDIA Pascal and Volta GPUs.
  • If dropping TensorRT 8.x support is a hard blocker for you, please comment on (#36306).

Bugfixes

• (Java) Fixed the Spark runner firing processing-time timers in reverse timestamp order (#39824).
• (Java) Fixed the Spark runner dropping the stored watermark of a streaming source with no update in a batch (#39822).
• (Python) Fixed incorrect profiler options handling on portable runners (#39613).
• (Java) KafkaIO dynamic reads no longer require the obsolete beam_fn_api experiment (#29998).
• (Prism) Self-checkpointing splittable DoFns now resume after their requested delay instead of immediately, so polling SDFs no longer busy-spin (#39848).
• (Java) MongoDbIO read splitting now preserves non-ObjectId _id types (e.g. string ids) instead of failing to parse the generated range filters (#39900).
• (Go) Fixed GCS glob matching silently dropping objects when the glob pattern contains multi-byte characters (#39969).
• (Python) Fixed TensorRTEngineHandlerNumPy failing with CUDA_ERROR_INVALID_VALUE on models with a single-element input or output tensor (#36306).
• (Python) Fixed PickleCoder/_MemoizingPickleCoder.as_deterministic_coder() raising TypeError instead of returning a working deterministic coder (#28558).

According to git shortlog, the following people contributed to the 2.77.0 release. Thank you to all contributors!

Abdelrahman Ibrahim, Aditya Narayan, Ahmed Abualsaud, Alex Bevilacqua, Alexander Pochill, Ali Ebrahim, Andrew Crites, Arun Pandian, Ashwin S, Bruno Volpato, Chamikara Jayalath, Chris Gavin, Claire McGinty, Danny McCormick, Derrick Williams, Eiji Ogiwara, Elia Liu, Fabian Loris, Goutam Adwant, HansMarcus01, Israel Herraiz, Jack McCluskey, Jan Lukavský, Jeremy Schoemaker, Kenneth Knowles, Lalit Yadav, Lawrence Qiu, M Junaid Shaukat, Makoto Nagai, Maksym Tymoshyk, Manvith Panyam, Mattie Fu, Michael Gruschke, Mukesh Bhandarkar, Nicolas Gibanel, Paulius Kuzmickas, Radosław Stankiewicz, Ryan Wigglesworth, Sam Whittle, Sharan Teja M, Shizuma5, Shunping Huang, SreeramaYeshwanthGowd, Tobias Kaymak, Tom Newton, Udit Jain, Vitaly Terentyev, Yi Hu, ZIHAN DAI, akshayjadiyanv, claudevdm, darshan-sj, feefs, junaiddshaukat, kellen, nitinware, parveensania, tvalentyn

We are happy to present the new 2.76.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.

For more information on changes in 2.76.0, check out the detailed release notes.

Highlights

• Added a full Iceberg batch and streaming changelog source (CDC) (#38831)
• (Java) Added per-element OpenTelemetry trace propagation across stages in the Dataflow Streaming Runner. Enable it with --experiments=enable_otel_defaults,element_metadata_supported,disable_portable_worker. Cloud Trace incurs additional cost. (#33176)
• (Java) Added OpenTelemetry header propagation support for both reads and writes in KafkaIO and PubSubIO. (#33176)
• (Java) Added OpenTelemetry tracing support for SpannerIO change streams (#33176)
• (Python) JmsIO (IBM MQ, ActiveMQ, and other providers) is now supported in Python via cross-language (#30716).

I/Os

• Upgraded Iceberg dependency to 1.11.0 (Java) (#38925).
• Add ArrowFlight IO (Java) (#20116).
• Added a Delta Lake batch changelog source (CDC) (#39492)

New Features / Improvements

• Added GroupIntoBatches transform and the standard
  beam:coder:sharded_key:v1 coder to the Go SDK, along with
  beam.Coder.IsDeterministic, beam.PCollection.WindowingStrategy,
  and coder.RegisterDeterministicCoder for opt-in deterministic
  custom coders (Go) (#19868).
• TriggerStateMachineRunner changes from BitSetCoder to SentinelBitSetCoder to
  encode finished bitset. SentinelBitSetCoder and BitSetCoder are state
  compatible. Both coders can decode encoded bytes from the other coder
  (#38139).
• (Python) Removed the envoy-data-plane (and transitive betterproto) dependency; EnvoyRateLimiter now uses a small vendored protobuf definition instead, resolving dependency conflicts for downstream projects (#37854).
• (Java) Supported acknowledge mode for JmsIO (#39253).
• (Python) Staged files directory is now automatically added to sys.path on the Python SDK worker at startup. This makes Python files provided via the '--files_to_stage' pipeline option importable in the pipeline code and makes it easier to initialize Python SDK harness at startup via the --beam_plugins pipeline option. For more information, see the Staging Individual Files section of the dependency management docs. This behavior can be disabled by passing the '--experiments=no_staged_dir_in_sys_path' pipeline option (#39431).
• (Python) Added equal_to_approx, an assert_that matcher that compares numeric pipeline outputs with a configurable tolerance (#18028).
• (Python) Timestamp now supports variable subsecond precision, up to nanoseconds. The portable
  beam:logical_type:timestamp:v1 logical type now maps to Python's Timestamp (#39344).
• (Python) Added UnboundedSource, an interface for reading an infinite stream of records with checkpointing, watermark reporting, and bundle finalization. Read one with beam.io.Read
  (#19137).
• (Python) Added Watch, a transform that polls a growing set of outputs for each input element, deduplicates outputs across poll rounds, and stops per a user-supplied termination condition
  (#21521).
• (Python) Added support to analyze core dumps created after python worker segmentation faults with pystack (or gdb if installed) using the --profiler_agent=coredump pipeline option. (#39484).
• (Python) Added Sample.Any, the Python equivalent of Java's Sample.any, which returns up to n arbitrary elements from a PCollection (#18552).

Breaking Changes

• (Python) Removed google-perftools from the SDK container images. Users who wish to use --profiler_agent=tcmalloc should install google-perftools APT package in their custom container images separately (#39323).

• [IcebergIO] Reading a timestamptz column will now return a Timestamp.MICROS Beam logical type to preserve
  microseconds (the old Beam Schema.FieldType#DATETIME primitive type truncates past milliseconds). This may break
  the following use cases when a timestamptz column is present:

  • Existing streaming read pipelines.
  • Managed Iceberg batch reads when upgraded from an older SDK.
  • Python reads.

  Use pipeline option --updateCompatibilityVersion=2.75.0 (or any older version) to keep the old behavior (#39344).

• DoFn.process returning a str, bytes, or dict (instead of an iterable wrapping one) now raises a TypeError rather than silently iterating per-character/byte/key (Python) (#18712).

• (Java) Added DRAINING and DRAINED states to PipelineResult, including runner state mappings and Dataflow update handling (#39020).

• (Python) Typehints of dataclass fields are honored during type inferences. To restore the behavior of fallback-to-any,
  use pipeline option --exclude_infer_dataclass_field_type (#38797).
  However fixing forward is recommended.

• (Java) IcebergIO and projects that use it must now be built with Java 17 or later as a result of Iceberg 1.11.0 upgrade (#38925).

Bugfixes

• Fixed unresolved runtime ValueProvider options being stringified in Python Dataflow Flex Templates (#39499).
• Fixed unbounded checkpoint state growth for splittable DoFns that self-checkpoint on the portable Flink runner (Java) (#27648).
• Improved Java pipeline performance by avoiding repeated DoFn type descriptor resolution when creating cached invokers (#39309).
• (Python) Fixed a memory leak in Python SDK caused by storing exceptions with potentially large stack frames in a cache (#39406).

Known Issues

• (Java) Projects using the Flink runner with Flink 2.1 or later alongside libraries requiring org.lz4:lz4-java (e.g., Kafka clients) may encounter a Gradle capability conflict, because Flink 2.1+ ships at.yawk.lz4:lz4-java which declares the same capability. To resolve, add a capabilitiesResolution rule to your build.gradle that selects at.yawk.lz4:lz4-java (#38947).

According to git shortlog, the following people contributed to the 2.76.0 release. Thank you to all contributors!

ADITYA RAJ, Abdelrahman Ibrahim, Ahmed Abualsaud, Alexander Kolb, Amar3tto, Andrew Crites, Arun Pandian, Aryankn29, Atharva Moroney, Avi Kondareddy, Bruno Volpato, Chamikara Jayalath, Chris Qiu, Claire McGinty, Danny McCormick, Derrick Williams, Elia Liu, Florian TREHAUT, Guflly, HansMarcus01, Ian Liao, Ivy Xu, Jack McCluskey, KRITI MITTAL, Kenneth Knowles, Lalit Yadav, Manvith Panyam, Minh Vu, Nikita Grover, PRADDZY, Peter Tran, Radosław Stankiewicz, Ryan Wigglesworth, Shahar Epstein, Shunping Huang, SreeramaYeshwanthGowd, Steven van Rossum, Tarun Annapareddy, Tejas Iyer, Tobias Kaymak, Tomasz Wojdat, Utkarsh Parekh, Venkata Bharath Malapati, Vitaly Terentyev, Yi Hu, ZIHAN DAI, aibrahiim, akshayjadiyanv, atognolas, claudevdm, janaom, jayjayakumar, raman118, shunping, tvalentyn

We are happy to present the new 2.75.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.

For more information on changes in 2.75.0, check out the detailed release notes.

Highlights

• Python SDK now supports memory profiling with Memray (#38853).
• (Python) Added Qdrant VectorDatabaseWriteConfig implementation (#38141).

I/Os

• Support for reading from Delta Lake added (Java) (#38551).
• ClickHouseIO: support writing DateTime64(precision[, 'timezone']) columns with sub-second precision (Java) (#38466).
• Upgraded IO Expansion Service to Java 17 (#38974).

New Features / Improvements

• Dataflow Runner v2 has been renamed to Dataflow Portable Runner. Please refer to Dataflow public documentation on when to enable Portable Runner.(#39000).
• (Java) Enabled state tag encoding v2 by default for new Dataflow Streaming Engine jobs. It can be disabled by passing --experiments=disable_streaming_engine_state_tag_encoding_v2 or --updateCompatibilityVersion=2.74.0 pipeline option. Note that the tag encoding version cannot change during a job update. Jobs using tag encoding v2 (enabled by default for new jobs on 2.75.0+) cannot be downgraded to Beam versions prior to 2.73.0, as only versions 2.73.0 and later support tag encoding v2. (#38705).
• (Python) Added instrumentation to support off-the-shelf profiling agents when launching Python SDK Harness (#38853).
• (Java) Added support to the FnApi Data stream protocol allowing runners to isolate bundles slowly processing input from other bundles. (#39001).
• (YAML) Switched js2py library to Quickjs (#38473).
• (YAML) Added HuggingFaceModelHandler for YAML usage (#38696).
• (YAML) Added WriteToMongoDB transform (#38376).
• (YAML) Added WriteToDatadog transform (#38362).
• (Java) Flink 2.1 and 2.2 support is added (#38947) (#38978); Flink 1.17 and 1.18 support is dropped.
• (Python) MqttIO is now supported in Python via cross-language (#21060).

Breaking Changes

• (Python) Typehints of dataclass fields are honored during type inferences. To restore the behavior of fallback-to-any,
  use pipeline option --exclude_infer_dataclass_field_type (#38797).
  However fixing forward is recommended.

Bugfixes

• Fixed GCS filesystem glob matching to correctly handle / in object names and support ** for recursive matching (Go) (#38059).
• Fixed BigQueryEnrichmentHandler batch mode dropping earlier requests when multiple requests share the same enrichment key (Python) (#38035).
• Fixed IcebergIO writing manifest column bounds padded with trailing 0x00 bytes, which broke equality predicate pushdown in some query engines (Java) (#38580).

Known Issues

• (Java) Projects using the Flink runner with Flink 2.1 or later alongside libraries requiring org.lz4:lz4-java (e.g., Kafka clients) may encounter a Gradle capability conflict, because Flink 2.1+ ships at.yawk.lz4:lz4-java which declares the same capability. To resolve, add a capabilitiesResolution rule to your build.gradle that selects at.yawk.lz4:lz4-java (#38947).

According to git shortlog, the following people contributed to the 2.75.0 release. Thank you to all contributors!

Abdelrahman Ibrahim, Ahmed Abualsaud, Akshat, Andrew Crites, Andrew Kabas, Anurag Pappula, Arpit Jain, Arun Pandian, Atharv, Chamikara Jayalath, Danny McCormick, Deji Ibrahim, Derrick Williams, Drew Stevens, Durgaprasad M L, Elia Liu, Enzo Maruffa Moreira, Ganesh Sivakumar, Goutam Adwant, HansMarcus01, Jack McCluskey, Joe Santos, Kenneth Knowles, Lalit Yadav, Liam Miller-Cushon, Maciej Szwaja, Manan Mangal, Michael Gruschke, Nikita Grover, Radek Stankiewicz, Radosław Stankiewicz, Reuven Lax, RuiLong J., Sachin Ranjalkar, Sagnik Ghosh, Sam Whittle, Shunping Huang, Subramanya V, Tarun Annapareddy, Tobias Kaymak, TongruiLi, Valentyn Tymofieiev, Vitaly Terentyev, XQ Hu, Yi Hu, aaaZayne, claudevdm, ddebowczyk92, innuendo, kellen, parveensania, tejasiyer-dev

We are happy to present the new 2.74.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.

For more information on changes in 2.74.0, check out the detailed release notes.

Highlights

• Spark 4 runner support for Java SDK (#38255).

I/Os

• IcebergIO: support declaring a table's sort order on dynamic table creation via the new sort_fields config (#38269).
• IcebergIO: support writing with hash distribution mode, and with autosharding (#38061).

New Features / Improvements

• Capability introduces an indicator for aggregations and timers firing during a pipeline drain, allowing users and sinks to recognize and appropriately handle potentially incomplete or partial data (#36884).
• Added support for setting disk provisioned IOPS and throughput in Dataflow runner via --diskProvisionedIops and --diskProvisionedThroughputMibps pipeline options (Java/Go/Python) (#38349).
• TriggerStateMachineRunner changes from BitSetCoder to SentinelBitSetCoder to
  encode finished bitset. SentinelBitSetCoder and BitSetCoder are state
  compatible. Both coders can decode encoded bytes from the other coder
  (#38139).
• (Python) Added type alias for with_exception_handling to be used for typehints. (#38173).
• (Java) BatchElements transform for Java SDK (#38369)
• Added plugin mechanism to support different Lineage implementations (Java) (#36790).
• (Python) Supported Python user type in Beam SQL. For example, SQL statements like SELECT some_field from PCOLLECTION can now operate a PCollection of Beam Row containing pickable Python user type (#20738).
• (Python) Introduced beam.coders.registry.register_row as preferred API to register a named tuple or dataclass with a Beam Row. At pipelne runtime, the original type associated with the registered row are preserved across the serialization boundary (#38108).

Breaking Changes

• (Python) Made Beartype the default fallback type checking tool. This can be disabled with the --disable_beartype pipeline option. (#38275)

Deprecations

• Dropped Java 8 support (#31678).
• Removed Samza Runner support (#35448).

Bugfixes

• Fixed BigQueryEnrichmentHandler batch mode dropping earlier requests when multiple requests share the same enrichment key (Python) (#38035).
• Added max_batch_duration_secs passthrough support in Python Enrichment BigQuery and CloudSQL handlers so batching duration can be forwarded to BatchElements (#38243).

According to git shortlog, the following people contributed to the 2.74.0 release. Thank you to all contributors!

Abdelrahman Ibrahim, Ahmed Abualsaud, Andrew Crites, Andrew Kabas, Arran Cudbard-Bell, Arun Pandian, Asish Kumar, Bentsi Leviav, Blake Jones, Bruno Volpato, Chris Jordan, Danny McCormick, Deji Ibrahim, Derrick Williams, Elia LIU, Ganesh Sivakumar, Jack McCluskey, Kenneth Knowles, Lalit Yadav, M Junaid Shaukat, Matej Aleksandrov, Prabhnoor Singh, Radek Stankiewicz, Radosław Stankiewicz, Reuven Lax, RuiLong J., Sam Whittle, Shunping Huang, Subramanya V, Tarun Annapareddy, Tobias Kaymak, TongruiLi, Valentyn Tymofieiev, Vitaly Terentyev, XQ Hu, Yi Hu, ZIHAN DAI, apanich, bambadiouf1, chenxuesdu, claudevdm, harshadkhetpal, johnjcasey, parveensania, tianz101

We are happy to present the new 2.73.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.

For more information on changes in 2.73.0, check out the detailed release notes.

Highlights

I/Os

• DebeziumIO (Java): added OffsetRetainer interface and FileSystemOffsetRetainer implementation to persist and restore CDC offsets across pipeline restarts, and exposed withStartOffset / withOffsetRetainer on DebeziumIO.Read and the cross-language ReadBuilder (#28248).

New Features / Improvements

• (Python) Added BigQuery CDC streaming source (#37724)
• Added ADKAgentModelHandler for running Google Agent Development Kit (ADK) agents (Python) (#37917).
• (Python) Added exception chaining to preserve error context in CloudSQLEnrichmentHandler, processes utilities, and core transforms (#37422).
• (Python) Added a pipeline option --experiments=pip_no_build_isolation to disable build isolation when installing dependencies in the runtime environment (#37331).
• (Go) Added OrderedListState support to the Go SDK stateful DoFn API (#37629).
• Added support for large pipeline options via a file (Python) (#37370).
• Supported infer schema from dataclass (Python) (#22085). Default coder for typehint-ed (or set with_output_type) for non-frozen dataclasses changed to RowCoder. To preserve the old behavior (fast primitive coder), explicitly register the type with FastPrimitiveCoder.
• Updates minimum Go version to 1.26.1 (#37897).
• (Python) Added image embedding support in apache_beam.ml.rag package (#37628).
• (Python) Added support for Python version 3.14 (#37247).

Breaking Changes

• The Python SDK container's boot.go now passes pipeline options through a file instead of the PIPELINE_OPTIONS environment variable. If a user pairs a new Python SDK container with an older SDK version (which does not support the file-based approach), the pipeline options will not be recognized and the pipeline will fail. Users must ensure their SDK and container versions are synchronized (#37370).
• Python DoFn.with_exception_handling now respects user DoFn typehints. This can break update compatibility if coders change. It can also break pipeline compilation if existing typehints are incorrect. To update safely sepcify the pipeline option --update_compatibility_version=2.72.0. To fix typehints replace any incorrect typehints that were previously ignored (#37590)

Bugfixes

• Fixed ProcessManager not reaping child processes, causing zombie process accumulation on long-running Flink deployments (Java) (#37930).

Security Fixes

• Fixed CVE-2023-46604 (CVSS 10.0) and CVE-2022-41678 by upgrading ActiveMQ from 5.14.5 to 5.19.2 (Java) (#37943).
• Fixed CVE-2024-1597, CVE-2022-31197, and CVE-2022-21724 by upgrading PostgreSQL JDBC Driver from 42.2.16 to 42.6.2 (Java) (#37942).

List of Contributors

According to git shortlog, the following people contributed to the 2.73.0 release. Thank you to all contributors!

Abdelrahman Ibrahim, Ahmed Abualsaud, Alex Malao, Alexander Nieuwenhuijse, Andres Tiko, Andrew Crites, Arun Pandian, Bentsi Leviav, Bruno Volpato, Chamikara Jayalath, Chandra Kiran Bolla, Danny McCormick, Deji Ibrahim, Derrick Williams, Elia LIU, Esmelealem, Hannes Gustafsson, Jack McCluskey, Joey Tran, Kenneth Knowles, M Junaid Shaukat, Mansi Singh, Matej Aleksandrov, Mathijs Deelen, Mattie Fu, Praneet Nadella, Radek Stankiewicz, Radosław Stankiewicz, Reuven Lax, RuiLong J., S. Veyrié, Sakthivel Subramanian, Sam Whittle, Shubham Thakur, Shunping Huang, Subramanya V, Tarun Annapareddy, Tobias Kaymak, Valentyn Tymofieiev, Vitaly Terentyev, XQ Hu, Yi Hu, ZIHAN DAI, claudevdm, kishorepola, parveensania

We are happy to present the new 2.72.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.

For more information on changes in 2.72.0, check out the detailed release notes.

Highlights

• Flink 2.0 support (#36947).

I/Os

• Add Datadog IO support (Java) (#37318).
• Remove Pubsublite IO support, since service will be deprecated in March 2026. (#37375).
• (Java) ClickHouse - migrating from the legacy JDBC driver (v0.6.3) to ClickHouse Java Client v2 (v0.9.6). See the class documentation for migration guide (#37610).
• (Java) Upgraded GoogleAdsIO to use GoogleAdsIO API v23 (#37620).

New Features / Improvements

• (Python) Added exception chaining to preserve error context in CloudSQLEnrichmentHandler, processes utilities, and core transforms (#37422).
• (Python) Added a pipeline option --experiments=pip_no_build_isolation to disable build isolation when installing dependencies in the runtime environment (#37331).

Deprecations

• (Python) Removed previously deprecated list_prefix method for filesystem interfaces (#37587).

Bugfixes

• Fixed (Yaml) issue with validate compatible method (#37588).
• Fixed (Yaml) issue with Create transform dealing with different type elements (#37585).

Security Fixes

• Fixed CVE-2024-28397 by switching from js2py to pythonmonkey (Yaml) (#37560).

List of Contributors

According to git shortlog, the following people contributed to the 2.72.0 release. Thank you to all contributors!

Abdelrahman Ibrahim, Ahmed Abualsaud, Andrew Crites, Arun Pandian, Ben Feinstein, Bentsi Leviav, Celeste Zeng, Danny McCormick, Danny Mccormick, Derrick Williams, Elia LIU, Ganesh, Jack McCluskey, Kenneth Knowles, Labesse Kévin, M Junaid Shaukat, Mansi Singh, Mattie Fu, Nayan Mathur, Pablo Estrada, Pirzada Ahmad Faraz, Radek Stankiewicz, Radosław Stankiewicz, Robert Bradshaw, Rohan Sah, RuiLong J., Sakthivel Subramanian, Sam Whittle, Shaheer Amjad, Shunping Huang, Steven van Rossum, Tarun Annapareddy, Tobias Kaymak, Valentyn Tymofieiev, Vitaly Terentyev, Yi Hu, XQ Hu, ZIHAN DAI, apanich, chenxuesdu, claudevdm, franzonia137

We are happy to present the new 2.71.0 release of Beam.
This release includes both improvements and new functionality.

For more information on changes in 2.71.0, check out the detailed release notes.

I/Os

• (Java) Elasticsearch 9 Support (#36491).
• (Java) Upgraded HCatalogIO to Hive 4.0.1 (#32189).

New Features / Improvements

• Support configuring Firestore database on ReadFn transforms (Java) (#36904).
• (Python) Inference args are now allowed in most model handlers, except where they are explicitly/intentionally disallowed (#37093).

Bugfixes

• Fixed FirestoreV1 Beam connectors allow configuring inconsistent project/database IDs between RPC requests and routing headers #36895 (Java) (#36895).
• Logical type and coder registry are saved for pipelines in the case of default pickler. This fixes a side effect of switching to cloudpickle as default pickler in Beam 2.65.0 (Python) (#35738).

Known Issues

For the most up to date list of known issues, see https://github.com/apache/beam/blob/master/CHANGES.md

List of Contributors

According to git shortlog, the following people contributed to the 2.71.0 release. Thank you to all contributors!

Abacn, Ahmed Abualsaud, Amar3tto, Andrew Crites, apanich, Arun, Arun Pandian, assaf127, Chamikara Jayalath, CherisPatelInfocusp, Cheskel Twersky, Claire McGinty, Claude, Danny Mccormick, dependabot[bot], Derrick Williams, Egbert van der Wal, Evan Galpin, Ganesh, github-actions[bot], hekk-kaori-maeda, Jack Dingilian, Jack McCluskey, JayajP, Jiang Zhu, Kenneth Knowles, liferoad, M Junaid Shaukat, Nayan Mathur, Noah Stapp, Paco Avila, Radek Stankiewicz, Radosław Stankiewicz, Robert Stupp, Sam Whittle, Shunping Huang, Steven van Rossum, Suvrat Acharya, Tarun Annapareddy, tvalentyn, Utkarsh Parekh, Vitaly Terentyev, Xiaochu Liu, Yala Huang Feng, Yi Hu, Yu Watanabe, zhan7236

We are happy to present the new 2.70.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.

For more information on changes in 2.70.0, check out the detailed release notes.

Highlights

• Flink 1.20 support added (#32647).

New Features / Improvements

• Python examples added for Milvus search enrichment handler on Beam Website
  including jupyter notebook example (Python) (#36176).
• Milvus sink I/O connector added (Python) (#36702).
  Now Beam has full support for Milvus integration including Milvus enrichment and sink operations.

Breaking Changes

• (Python) Some Python dependencies have been split out into extras. To ensure all previously installed dependencies are installed, when installing Beam you can pip install apache-beam[gcp,interactive,yaml,redis,hadoop,tfrecord], though most users will not need all of these extras (#34554).

Deprecations

• (Python) Python 3.9 reached EOL in October 2025 and support for the language version has been removed. (#36665).

List of Contributors

According to git shortlog, the following people contributed to the 2.70.0 release. Thank you to all contributors!

Abdelrahman Ibrahim, Ahmed Abualsaud, Alex Chermenin, Andrew Crites, Arun Pandian, Celeste Zeng, Chamikara Jayalath, Chenzo, Claire McGinty, Danny McCormick, Derrick Williams, Dustin Rhodes, Enrique Calderon, Ian Liao, Jack McCluskey, Jessica Hsiao, Joey Tran, Karthik Talluri, Kenneth Knowles, Maciej Szwaja, Mehdi.D, Mohamed Awnallah, Praneet Nadella, Radek Stankiewicz, Radosław Stankiewicz, Reuven Lax, RuiLong J., S. Veyrié, Sam Whittle, Shunping Huang, Stephan Hoyer, Steven van Rossum, Tanu Sharma, Tarun Annapareddy, Tom Stepp, Valentyn Tymofieiev, Vitaly Terentyev, XQ Hu, Yi Hu, changliiu, claudevdm, fozzie15, kristynsmith, wolfchris-google

We are happy to present the new 2.69.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.

For more information on changes in 2.69.0, check out the detailed release notes.

Highlights

• (Python) Add YAML Editor and Visualization Panel (#35772).
• (Java) Java 25 Support (#35772).

I/Os

• Upgraded Iceberg dependency to 1.10.0 (#36123).

New Features / Improvements

• Enhance JAXBCoder with XMLInputFactory support (Java) (#36446).
• Python examples added for CloudSQL enrichment handler on Beam website (Python) (#35473).
• Support for batch mode execution in WriteToPubSub transform added (Python) (#35990).
• Added official support for Python 3.13 (#34869).
• Added an optional output_schema verification to all YAML transforms (#35952).
• Support for encryption when using GroupByKey added, along with --gbek pipeline option to automatically replace all GroupByKey transforms (Java/Python) (#36214).

Breaking Changes

• (Python) dill is no longer a required, default dependency for Apache Beam (#21298).
  • This change only affects pipelines that explicitly use the pickle_library=dill pipeline option.
  • While dill==0.3.1.1 is still pre-installed on the official Beam SDK base images, it is no longer a direct dependency of the apache-beam Python package. This means it can be overridden by other dependencies in your environment.
  • If your pipeline uses pickle_library=dill, you must manually ensure dill==0.3.1.1 is installed in both your submission and runtime environments.
    • Submission environment: Install the dill extra in your local environment pip install apache-beam[gcpdill].
    • Runtime (worker) environment: Your action depends on how you manage your worker's environment.
      • If using default containers or custom containers with the official Beam base image e.g. FROM apache/beam_python3.10_sdk:2.69.0
        • Add dill==0.3.1.1 to your worker's requirements file (e.g., requirements.txt)
        • Pass this file to your pipeline using the --requirements_file requirements.txt pipeline option (For more details see managing Dataflow dependencies).
      • If custom containers with a non-Beam base image e.g. FROM python:3.9-slim
        • Install apache-beam with the dill extra in your docker file e.g. RUN pip install --no-cache-dir apache-beam[gcp,dill]
  • If there is a dill version mismatch between submission and runtime environments you might encounter unpickling errors like Can't get attribute '_create_code' on <module 'dill._dill' from....
  • If dill is not installed in the runtime environment you will see the error ImportError: Pipeline option pickle_library=dill is set, but dill is not installed...
  • Report any issues you encounter when using pickle_library=dill to the GitHub issue (#21298)
• (Python) Added a pickle_library=dill_unsafe pipeline option. This allows overriding dill==0.3.1.1 using dill as the pickle_library. Use with extreme caution. Other versions of dill has not been tested with Apache Beam (#21298).
• (Python) The deterministic fallback coder for complex types like NamedTuple, Enum, and dataclasses now normalizes filepaths for better determinism guarantees. This affects streaming pipelines updating from 2.68 to 2.69 that utilize this fallback coder. If your pipeline is affected, you may see a warning like: "Using fallback deterministic coder for type X...". To update safely sepcify the pipeline option --update_compatibility_version=2.68.0 (#36345).
• (Python) Fixed transform naming conflict when executing DataTransform on a dictionary of PColls (#30445).
  This may break update compatibility if you don't provide a --transform_name_mapping.
• Removed deprecated Hadoop versions (2.10.2 and 3.2.4) that are no longer supported for Iceberg from IcebergIO (#36282).
• (Go) Coder construction on SDK side is more faithful to the specs from runners without stripping length-prefix. This may break streaming pipeline update as the underlying coder could be changed (#36387).
• Minimum Go version for Beam Go updated to 1.25.2 (#36461).
• (Java) DoFn OutputReceiver now requires implementing a builder method as part of extended metadata support for elements (#34902).
• (Java) Removed ProcessContext outputWindowedValue introduced in 2.68 that allowed setting offset and record Id. Use OutputReceiver's builder to set those field ([#36523]#36523).

Bugfixes

• Fixed passing of pipeline options to x-lang transforms when called from the Java SDK (Java) (#36443).
• PulsarIO has now changed support status from incomplete to experimental. Both read and writes should now minimally
  function (un-partitioned topics, without schema support, timestamp ordered messages for read) (Java)
  (#36141).
• Fixed Spanner Change Stream reading stuck issue due to watermark of partition moving backwards (#36470).

List of Contributors

According to git shortlog, the following people contributed to the 2.69.0 release. Thank you to all contributors!

Abdelrahman Ibrahim, Ahmed Abualsaud, Andrew Crites, Arun Pandian, Bryan Dang, Chamikara Jayalath, Charles Nguyen, Chenzo, Clay Johnson, Danny McCormick, David A, Derrick Williams, Enrique Calderon, Hai Joey Tran, Ian Liao, Ian Mburu, Jack McCluskey, Jiang Zhu, Joey Tran, Kenneth Knowles, Kyle Stanley, Maciej Szwaja, Minbo Bae, Mohamed Awnallah, Radek Stankiewicz, Radosław Stankiewicz, Razvan Culea, Reuven Lax, Sagnik Ghosh, Sam Whittle, Shunping Huang, Steven van Rossum, Talat UYARER, Tanu Sharma, Tarun Annapareddy, Tom Stepp, Valentyn Tymofieiev, Vitaly Terentyev, XQ Hu, Yi Hu, Yilei, claudevdm, flpablo, fozzie15, johnjcasey, lim1t, parveensania, yashu