[Tianlei Wu (tianleiwu)]

Description

• Add the CUDA GatedDeltaNet contrib operator with recurrent decode and tensor-core chunked prefill paths.
• Support ragged and rank-4 inputs, fused Qwen gate normalization, and native GDN arithmetic from raw A_log.
• Return the compact three-output ABI [output, final_state, state_update]. The FP32 state_update capsule packs decay, shared-key, and delta transitions needed to replay accepted speculative prefixes without materializing dense recurrent checkpoints.
• Add schema and shape validation, CUDA registration, focused tests, a microbenchmark, and an authored operator guide.

Motivation and Context

Dense recurrent checkpoints scale the full FP32 state with draft width. At the Qwen3.8 geometry, a four-slot window across 48 GDN layers consumes 576 MiB. Compact transition capture keeps one committed state and records only the information needed to reconstruct an accepted prefix.

The schema intentionally exposes only native arithmetic and does not include the experimental arithmetic_mode attribute. Models exported with the retired experimental ABI must be re-exported.

Companion ONNX Runtime GenAI integration: microsoft/onnxruntime-genai#2472

Validation

• ./build/cu130/Debug/onnxruntime_provider_test --gtest_filter='*GatedDeltaNet*'
• 26/26 focused tests passed on H200 with CUDA 13.0 and cuDNN 9.23.

Performance and Quality

On H200, context 2048, generation 256, with five paired fresh-process repetitions per batch:

• Native arithmetic won all 40 MTP and DFlash2 throughput pairs versus the retired compatibility experiment. Median native/compatibility ratios were 1.0519 for MTP and 1.0543 for DFlash2.
• Paired quality differences were not statistically significant: MMLU-Pro 83.75% native vs. 83.38% compatibility (McNemar p=0.73); GPQA 81.82% vs. 78.28% (p=0.23).
• Against the retired separate-factor representation, the packed capsule preserved exact tokens and replay work. It won 20/20 MTP pairs and was at parity for DFlash2 batch 16, with a small-batch benefit.

[Copilot]

Pull request overview

Adds CUDA GatedDeltaNet and compact state updates, extends paged attention and quantized GEMV paths for speculative decoding, and adds supporting tests and documentation.

Changes:

• Adds GatedDeltaNet and packed variable-length causal convolution.
• Extends paged attention with non-causal FlashAttention support.
• Expands small-M and quantized GEMV kernels to 64 rows.

Reviewed changes

Copilot reviewed 42 out of 42 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
onnxruntime/core/graph/contrib_ops/bert_defs.cc	Adds schemas and causality attribute.
onnxruntime/core/graph/contrib_ops/ms_opset.h	Registers new schemas.
onnxruntime/contrib_ops/cuda/cuda_contrib_kernels.cc	Registers CUDA kernels.
onnxruntime/contrib_ops/cuda/bert/gated_delta_net.h	Declares GDN kernel.
onnxruntime/contrib_ops/cuda/bert/gated_delta_net.cc	Validates and dispatches GDN.
onnxruntime/contrib_ops/cuda/bert/gated_delta_net_impl.h	Defines GDN launch interface.
onnxruntime/contrib_ops/cuda/bert/gated_delta_net_impl.cu	Implements recurrent/chunked GDN.
onnxruntime/contrib_ops/cuda/bert/gated_delta_net_mma.cuh	Adds tensor-core primitives.
onnxruntime/contrib_ops/cuda/bert/gated_delta_net_plan.h	Defines planning heuristics.
onnxruntime/contrib_ops/cuda/bert/gated_delta_net_plan.cc	Implements plan caching.
onnxruntime/contrib_ops/cuda/bert/gated_delta_net_split_impl.cu	Adds split chunked engine.
onnxruntime/contrib_ops/cuda/bert/varlen_causal_conv_with_state.h	Declares packed convolution.
onnxruntime/contrib_ops/cuda/bert/varlen_causal_conv_with_state.cc	Validates packed convolution inputs.
onnxruntime/contrib_ops/cuda/bert/varlen_causal_conv_with_state_impl.cu	Implements packed convolution.
onnxruntime/contrib_ops/cuda/bert/paged_attention.h	Stores causality configuration.
onnxruntime/contrib_ops/cuda/bert/paged_attention.cc	Routes non-causal attention.
onnxruntime/contrib_ops/cuda/bert/paged_attention_impl.cu	Passes causality to FlashAttention.
onnxruntime/contrib_ops/cuda/bert/group_query_attention.cc	Allows batched subsequent prompts.
onnxruntime/contrib_ops/cuda/bert/causal_conv_with_state.cc	Improves validation context.
onnxruntime/contrib_ops/cpu/bert/group_query_attention_helper.h	Adds prompt-validation override.
onnxruntime/contrib_ops/cpu/bert/causal_conv_with_state.cc	Supplies operator name to validation.
onnxruntime/contrib_ops/cpu/bert/causal_conv_with_state_helper.h	Generalizes validation messages.
onnxruntime/contrib_ops/cpu/bert/attention_parameters.h	Adds paged-attention causality state.
onnxruntime/core/providers/cuda/math/matmul.cc	Moves counter initialization into launcher.
onnxruntime/core/providers/cuda/math/matmul_small_n_gemv.h	Documents 64-row support.
onnxruntime/core/providers/cuda/math/matmul_small_n_gemv.cu	Adds chunked row launches.
onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp8.h	Exposes FP8 GEMV limit.
onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp8.cc	Uses dynamic FP8 GEMV limit.
onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp8.cu	Adds multi-row FP8 MMA.
onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp4.h	Exposes FP4 GEMV limit.
onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp4.cc	Uses dynamic FP4 GEMV limit.
onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp4.cu	Adds multi-row FP4 MMA.
onnxruntime/python/tools/microbench/gdn.py	Adds GDN microbenchmark.
onnxruntime/test/python/transformers/test_paged_attention.py	Tests non-causal paged attention.
onnxruntime/test/providers/cuda/test_cases/matmul_small_n_gemv_op_test.cc	Tests expanded GEMV shapes.
onnxruntime/test/contrib_ops/matmul_block_scaled_fp8_test.cc	Tests FP8 row tiling.
onnxruntime/test/contrib_ops/matmul_block_scaled_fp4_test.cc	Tests FP4 row tiling.
onnxruntime/test/contrib_ops/gated_delta_net_op_test.cc	Covers GDN behavior and planning.
onnxruntime/test/contrib_ops/causal_conv_with_state_op_test.cc	Covers packed convolution.
docs/contrib_ops/cuda/gdn.md	Documents GDN contracts and results.
docs/ContribOperators.md	Updates generated schemas.
docs/OperatorKernels.md	Updates generated kernel inventory.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Baiju Meswani (baijumeswani)]

Looks good.