[Tianlei Wu (tianleiwu)]

Expand compact fpA_intB MatMulNBits support to BF16 activations while retaining FP16 support for INT4 and INT8 weights with block_size=32. This enables the compact CUDA path for BF16 workloads on SM80 and newer GPUs, adds the matching dispatch and eligibility coverage, and documents the supported contract.

Summary of Changes

Compact CUDA support

• Extend compact USE_COMPACT_FPA_INTB_GEMM GEMM and GEMV dispatch to BF16 activations for INT4 and INT8 weights.
• Require SM80 or newer for BF16 compact kernels while preserving the existing FP16 behavior.
• Add BF16 profiler mappings, prepack diagnostics, workspace eligibility handling, and architecture checks.
• Keep INT2 outside the compact fpA_intB contract.

Build and documentation

File	Change
cmake/CMakeLists.txt	Update compact fpA_intB capability descriptions for FP16/BF16 and INT4/INT8.
cmake/onnxruntime_cuda_source_filters.cmake	Retain compact FP16/BF16 GEMM and GEMV sources.
docs/contrib_ops/cuda/matmul_nbits.md	Document compact activation types, weight widths, block size, and architecture requirements.

Tests

• Add BF16 compact INT4/INT8 kernel coverage and SM80 support checks.
• Extend MatMulNBits workspace eligibility tests for BF16 and SM75/SM80 behavior.

Testing

• lintrunner -a passed with Ruff and ClangFormat.
• Built and linked onnxruntime_provider_test in build/cu130_fp4_bench/Release.
• Ran 8 CUDA MatMulNBits runtime tests on an NVIDIA H200 (SM90); all passed, including FP16/BF16 INT4/INT8, BF16 small-M, no-zero-point, and bias cases.
• git diff --check passed.
• The standalone CUDA internal-test module was not built because onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=OFF in the local build configuration.

Motivation and Context

The compact fpA_intB implementation previously handled FP16 activations only. Adding BF16 support at the same block-size and weight-width contract allows the existing CUDA execution path to serve BF16 models without changing the INT2 behavior or requiring a separate fallback-specific implementation.

Checklist

• Tests added/updated
• No breaking changes
• Documentation updated

[Copilot]

Copilot review overview

🟡 Changes recommended

The prepacked-weight diagnostic incorrectly claims BF16 support begins at SM75 instead of SM80.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1

Open (1)

• Correct BF16 prepacked model compute capability requirements · New

What changed in this PR

Extends compact CUDA MatMulNBits to support BF16 activations with INT4/INT8 weights on SM80+.

Changes:

• Adds BF16 eligibility, GEMM/GEMV dispatch, and profiler mappings.
• Retains BF16 kernel sources in compact builds.
• Adds coverage and documents hardware constraints.

File	Description
cmake/​CMakeLists.txt	Updates compact-kernel build descriptions.
cmake/​onnxruntime_cuda_source_filters.cmake	Retains compact BF16 kernel sources.
docs/​contrib_ops/​cuda/​matmul_nbits.md	Documents BF16 support and SM80 requirement.
onnxruntime/​contrib_ops/​cuda/​llm/​fpA_intB_gemv/​fpA_intB_gemv.cu	Adds BF16 GEMV dispatch and validation.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_nbits.cc	Adds BF16 eligibility and GEMM profiling.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_nbits.h	Updates prepacked-weight diagnostics.
onnxruntime/​test/​contrib_ops/​cuda_kernels/​fpA_intB_gemm_kernel_test.cc	Adds compact BF16 kernel tests.
onnxruntime/​test/​providers/​cuda/​test_cases/​matmul_nbits_workspace_test.cc	Tests BF16 eligibility and architecture gating.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.