Skip to content

Add group size 6 support to paged XQA - #32108

Merged
Baiju Meswani (baijumeswani) merged 1 commit into
mainfrom
baijumeswani/paged-xqa-group6
Aug 16, 2026
Merged

Baiju Meswani (baijumeswani) merged 1 commit into
mainfrom
baijumeswani/paged-xqa-group6

Conversation

@baijumeswani

Copy link
Copy Markdown
Contributor

Enable the optimized xqa decode kernel for group size 6 when there are 6 kv heads per query

Copilot AI balanced review requested due to automatic review settings August 15, 2026 05:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds group-size-6 support to the optimized CUDA paged XQA decode path.

Changes:

  • Instantiates and dispatches group-size-6 XQA kernels.
  • Adds parity and dispatch telemetry tests.
  • Documents eligibility and shared-memory fallback behavior.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file
File Description
onnxruntime/test/python/transformers/test_paged_attention.py Adds group-size-6 dispatch tests and GPU capability checks.
onnxruntime/contrib_ops/cuda/bert/xqa/xqa_paged_loader.h Updates supported-group and shared-memory contracts.
onnxruntime/contrib_ops/cuda/bert/xqa/xqa_paged_loader.cu Updates kernel-instantiation documentation.
onnxruntime/contrib_ops/cuda/bert/xqa/xqa_paged_loader_impl.cuh Instantiates and dispatches group-size-6 kernels.
onnxruntime/contrib_ops/cuda/bert/paged_attention.cc Enables group size 6 and strengthens fallback handling.
docs/contrib_ops/cuda/paged_attention.md Documents paged XQA behavior and requirements.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@tianleiwu Tianlei Wu (tianleiwu) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the paged XQA group-size-6 specialization, runtime eligibility/fallback logic, shared-memory gating, documentation, and dispatch/parity coverage. The new specialization follows the existing padded-row kernel structure used by groups 4 and 8, and the group-specific shared-memory query matches the dispatched kernel.

Validation on H200: the CUDA provider and Python binding targets built successfully; both new group-size-6 dispatch tests passed, and the full paged-XQA class passed 29/31 cases. The remaining two FP8 harness failures occur before ORT execution because this local PyTorch cannot convert float8 tensors directly to NumPy; the parent commit has the same conversion path.

No PR-specific concerns found.

@baijumeswani
Baiju Meswani (baijumeswani) merged commit e858164 into main Aug 16, 2026
88 checks passed
@baijumeswani
Baiju Meswani (baijumeswani) deleted the baijumeswani/paged-xqa-group6 branch August 16, 2026 01:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

⌂ 换网址