Repository navigation
Add group size 6 support to paged XQA - #32108
Conversation
There was a problem hiding this comment.
Pull request overview
Adds group-size-6 support to the optimized CUDA paged XQA decode path.
Changes:
- Instantiates and dispatches group-size-6 XQA kernels.
- Adds parity and dispatch telemetry tests.
- Documents eligibility and shared-memory fallback behavior.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
onnxruntime/test/python/transformers/test_paged_attention.py |
Adds group-size-6 dispatch tests and GPU capability checks. |
onnxruntime/contrib_ops/cuda/bert/xqa/xqa_paged_loader.h |
Updates supported-group and shared-memory contracts. |
onnxruntime/contrib_ops/cuda/bert/xqa/xqa_paged_loader.cu |
Updates kernel-instantiation documentation. |
onnxruntime/contrib_ops/cuda/bert/xqa/xqa_paged_loader_impl.cuh |
Instantiates and dispatches group-size-6 kernels. |
onnxruntime/contrib_ops/cuda/bert/paged_attention.cc |
Enables group size 6 and strengthens fallback handling. |
docs/contrib_ops/cuda/paged_attention.md |
Documents paged XQA behavior and requirements. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Tianlei Wu (tianleiwu)
left a comment
There was a problem hiding this comment.
Reviewed the paged XQA group-size-6 specialization, runtime eligibility/fallback logic, shared-memory gating, documentation, and dispatch/parity coverage. The new specialization follows the existing padded-row kernel structure used by groups 4 and 8, and the group-specific shared-memory query matches the dispatched kernel.
Validation on H200: the CUDA provider and Python binding targets built successfully; both new group-size-6 dispatch tests passed, and the full paged-XQA class passed 29/31 cases. The remaining two FP8 harness failures occur before ORT execution because this local PyTorch cannot convert float8 tensors directly to NumPy; the parent commit has the same conversion path.
No PR-specific concerns found.
Enable the optimized xqa decode kernel for group size 6 when there are 6 kv heads per query