In the ZYFA vs AITER FP8 attention comparison, ZYFA wins on both speed and accuracy for causal prefill on AMD Instinct MI355X GPUs. Zyphra released it under Apache 2.0 on November 6, 2026, and its launch post reports 1.16 to 1.76 times the throughput of AITER's default FP8 kernels on real model activations, with 62 to 78% lower output error at head dimension 128. If you serve long prompts on MI355X, ZYFA is the better prefill kernel. If you also need decode, paged KV caches or variable-length batches, AITER stays in your stack because ZYFA does not cover them. The code is on GitHub and the method is described in the technical paper.
TL;DR
- ZYFA is Zyphra's open-source (Apache 2.0) prefill attention engine for AMD MI355X (gfx950), in BF16 and FP8.
- On activations captured from Mistral-7B, Qwen3-8B and DeepSeek-V2-Lite at 16K tokens, ZYFA FP8 reaches 1.16 to 1.76 times AITER's FP8 throughput.
- At head dimension 128, ZYFA's FP8 error is 62 to 78% lower than AITER's, and close to Composable Kernel (CK) and Triton.
- In BF16 the two are effectively tied: ZYFA stays within 3% of AITER at 16K and 32K tokens (Zyphra), and edges ahead by up to 7% at 2K.
- Scope is narrow: no decoding, no paged KV caches, no variable-length batches. Keep AITER for those.
- Installation is one pip command, and the Python call matches
flash_attn.flash_attn_func.
What is ZYFA and who built it?
ZYFA stands for Zyphra FlashAttention on AMD. It is the attention engine inside the Zyphra Optimization Engine (ZOE), which runs Zyphra Cloud, so an inference operator has published the kernel it runs in production. The authors are Quentin Anthony, Ganesh Bikshandi, Praneeth Medepalli and Stephen Ebert. The kernels target the MI355X only, per the project's repository; the BF16 kernels are written in HIP, AMD's C++ kernel language. The FP8 kernels are hand-scheduled gfx950 assembly produced by a Python generator, according to the project's repository.
AITER is AMD's library of inference kernels, published at github.com/ROCm/aiter. It is the default reference most ROCm users reach for, and it handles both prefill and decode.
How fast is ZYFA vs AITER FP8 attention?
Zyphra measured on real activations from one layer each of Mistral-7B, Qwen3-8B and DeepSeek-V2-Lite at 16K tokens, timed at sustained clocks. ZYFA's FP8 kernels reached 1.16 to 1.76 times AITER's default FP8 throughput (launch post, Figure 1).
On Gaussian synthetic inputs between 2K and 32K tokens the gap widens to 1.26 to 2.02 times. Synthetic data is the friendlier test, so treat the real-activation range as the more realistic expectation.
| Kernel | Source | FP8 prefill speed | FP8 error (head dim 128) | Covers decode and paged KV |
|---|---|---|---|---|
| ZYFA | Apache 2.0, Zyphra | 1.16 to 1.76x AITER on real activations 1 | 62 to 78% lower than AITER 1 | No |
| AITER | AMD, MIT licence | Baseline | Baseline | Yes |
| CK / Triton | AMD baselines | Not the throughput leaders | Within a few percent of ZYFA | Varies |
Figures are from the launch post. They come from one MI355X node at a single point in time.
Why does AITER's FP8 attention fall short of 2x?
FP8 matrix instructions offer twice the peak throughput of BF16, but attention kernels rarely get close to a 2x gain. At 32K tokens, AITER's FP8 kernel reaches 39% of FP8 peak, while its BF16 kernel reaches 49% of BF16 peak (Zyphra).
The reason is structural. FP8 accelerates only the two matrix multiplications in attention. The softmax between them runs at the same speed as before, so it becomes the bottleneck once the matrix work shrinks. A kernel that treats FP8 as a drop-in dtype swap leaves most of the hardware gain unused.
How does ZYFA get faster and more accurate at once?
Zyphra describes three changes in its paper:
- Lean P. The exponential in the softmax is replaced by a single multiply-add. This raises the error of one attention call to 1.22 to 1.36 times the exact-P error. Even so, across 68 captured layers the geometric-mean error is 0.26 times AITER's.
- A two-wave ping-pong schedule, which overlaps the matrix work of one group of waves with the softmax work of another.
- Per-block scaling with normalisation by the rounded row sum, which keeps small probabilities representable in FP8.
Speed and accuracy are not in tension here because AITER's FP8 path loses precision in places that ZYFA handles explicitly. Outputs are also bitwise identical from run to run, and the kernels work under torch.compile and in CUDA graphs.
Should you replace AITER with ZYFA?
Replace it for prefill only. Long-context prompt processing on MI355X is where ZYFA applies and where the measured gains sit. For decode, paged KV caches and variable-length batching, keep AITER, since ZYFA does not implement them. ZYFA supports head dimension 128 and the multi-head latent attention shape used by DeepSeek-V2, with query-key dimension 192 and value dimension 128.
If your workload is BF16, there is little reason to move. Measured on the same MI355X node, ZYFA's BF16 kernels land within 3% of AITER's at 16K and 32K tokens, and run up to 7% faster at 2K (Zyphra).
How do you install and use ZYFA?
The repository README lists the following requirements: an MI355X, a ROCm build of PyTorch, and testing on ROCm 7.2.3 with PyTorch 2.12.
git clone https://github.com/Zyphra/zyfa
cd zyfa
pip install . --no-build-isolation
Keep --no-build-isolation. The build compiles against your installed PyTorch, and the assembler looks for the ROCm compiler under $ROCM_PATH, which defaults to /opt/rocm.
Then swap the prefill call. The interface takes the same arguments as flash_attn.flash_attn_func:
from zyfa import flash_attn_func, fp8_quantize
# One power-of-two scale per tensor
(q8, q_s), (k8, k_s), (v8, v_s) = fp8_quantize(q), fp8_quantize(k), fp8_quantize(v)
out = flash_attn_func(q8, k8, v8, causal=True) # BF16 output
Leave decode and paged-KV paths on AITER.
What are the limits of these results?
Treat the numbers as a strong first result, not a settled benchmark: one MI355X node, one point in time, causal prefill only. The real-activation tests use three models, one layer each. Your model, sequence mix and serving stack may land elsewhere in the 1.16 to 1.76 times range. Run your own prefill benchmark on representative prompts before changing production, and check output quality on your own evaluations, since FP8 error is workload dependent.
Where does this fit in the wider open-source picture?
ZYFA follows the pattern of an infrastructure operator open-sourcing the component that carries its own traffic, shared by Kimi K3.1 Pro on the weights side and Ripple 65B against open-weight coding models from a chip lab. For inference cost context, see cheaper subagent status updates in Claude Code; for another throughput comparison, Celeris 1 Magnus against GPT-5.6 Sol; and for the self-hosting logic, Postgres for everything.
FAQ
Q: Is ZYFA faster than AITER for FP8 attention?
A: Yes, for causal prefill on MI355X: Zyphra reports 1.16 to 1.76 times AITER's FP8 throughput on real activations, and 1.26 to 2.02 times on synthetic inputs.
Q: Is ZYFA's FP8 output less accurate because it is faster?
A: No. At head dimension 128, its FP8 error is 62 to 78% lower than AITER's on Mistral-7B and Qwen3-8B activations, and within a few percent of CK and Triton kernels.
Q: Does ZYFA support decoding or paged KV caches?
A: No. It covers causal forward prefill only, with no decoding, paged KV caches or variable-length batches. Keep AITER for those paths.
Q: Which GPUs and software does ZYFA need?
A: An AMD Instinct MI355X (gfx950) and a ROCm build of PyTorch; the README lists ROCm 7.2.3 with PyTorch 2.12 as the tested pair.
Q: What licence is ZYFA released under?
A: Apache 2.0, with source code on GitHub at Zyphra/zyfa and a technical paper on arXiv.
Q: Is ZYFA better than AITER in BF16?
A: Not meaningfully. ZYFA's BF16 kernels are within 3% of AITER's at 16K and 32K tokens, and up to 7% higher at 2K, per Zyphra's measurements.
Top comments (0)