fla-org / flash-linear-attention
← 返回首页
🚀 Efficient implementations for emerging model architectures
📄 仓库 README
在 GitHub 查看 ↗来自 GitHub 仓库自述文件 · 版权归原作者所有
[](https://huggingface.co/fla-hub) [](https://discord.gg/vDaJTmKNcS)
💥 Flash Linear Attention brings together hardware-efficient building blocks, training-ready layers, and components for modern sequence models, spanning linear attention, sparse attention, state space models, and hybrid LLM architectures. All implementations are platform-agnostic and verified on NVIDIA, AMD, and Intel hardware. Pull requests are welcome!
- News
- Models
- Installation
- Usage
- Token Mixing
- Fused Modules
- Generation
- Hybrid Models
- Training
- Evaluation
- Benchmarks
- Citation
- Acknowledgements
News
- [2026-10] 🌊 Add CyFA (Cyclic Flow Attention) implementation to
fla- a Linear RNN that partitions memory by relative time through learned cyclic transport. - [2026-10] 🎉 Add Stick-Breaking Attention kernels and models to
fla(paper). - [2026-07] 🐈 Add CAT (Compress and Attend Transformer) implementation to
fla(paper) - a meta-sequence mixer that unlocks test-time control of inference costs. - [2026-07] 🧱 Add a Gluon backend for AttnRes.
- [2026-07] 🚀 Add FlashQLA backend for Gated DeltaNet.
- [2026-06] 🔭 Add Parallax implementation to
fla(paper). - [2026-06] 🧮 Add Preconditioned Gated DeltaNet (PGDN) and Preconditioned KDA (PKDA) to
fla(paper) — curvature-aware preconditioning of the linear recurrence via an ATK preconditioner. - [2026-06] 🧱 Add Wall attention implementation to
fla(blog). - [2026-05] 🚪 Add Gated DeltaNet 2 (GDN-2) implementation to
fla(paper). - [2026-05] 🦅 Add Raven implementation to
fla(repo). - [2026-05] 🚀 Add YOCO (You Only Cache Once) implementation to
fla. - [2026-05] ⚡ Add fused AttnRes support to
fla(paper). - [2026-04] 🐍 Add Mamba3 implementation to
fla(paper). - [2026-04] 🧱 Add MoBA (Mixture of Block Attention) implementation to
fla, with FlashMoBA backend support. - [2026-04] 🧱 Add TileLang backend support for selected kernels.
- [2026-04] 🎯 Add GPT-OSS-style attention sink support to
fla's attention kernels. - [2026-03] 🚀 Add Context Parallel support for KDA and GDN, enabling efficient distributed training across sequence dimension.
Older news
- [2025-10] 🌘 Add Kimi Delta Attention (KDA) implementation to `fla` ([paper](https://arxiv.org/abs/2510.26692)). - [2025-09] 🌲 Add DeltaFormer implementation to `fla` ([paper](https://arxiv.org/abs/2505.19488v1)). - [2025-09] 🐻 Thrilled to announce that [GDN](https://github.com/fla-org/flash-linear-attention/blob/HEAD/fla/ops/gated_delta_rule) has been integrated into Qwen3-Next. Check out their [blog post](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd&from=research.latest-advancements-list) for more info! - [2025-08] 🌲 Add Log-Linear Attention implementation to `fla` ([paper](https://arxiv.org/abs/2506.04761)). - [2025-08] 🎓 Add MoM implementation to `fla` ([paper](https://arxiv.org/abs/2502.13685)). - [2025-07] 🐳 Add MLA implementation to `fla` ([paper](https://arxiv.org/abs/2405.04434)). - [2025-07] 🛣️ Add PaTH Attention implementation to `fla` ([paper](https://arxiv.org/abs/2505.16381)). - [2025-06] 🎉 Add MesaNet implementation to `fla` ([paper](https://arxiv.org/abs/2506.05233)). - [2025-06] 🐍 Add Comba implementation to `fla` ([paper](https://arxiv.org/abs/2506.02475)). - [2025-05] 🎉 Add Rodimus* implementation to `fla` ([paper](https://arxiv.org/abs/2410.06577)). - [2025-04] 🎉 Add DeltaProduct implementation to `fla` ([paper](https://arxiv.org/abs/2502.10297)). - [2025-04] 🎉 Add FoX implementation to `fla` ([paper](https://arxiv.org/abs/2503.02130)). - [2025-03] ~~We have changed the default `initializer_range` to the magic 🐳 0.006~~ The `initializer_range` was rolled back to the default value of 0.02. For actual training, we recommend trying both. - [2025-02] 🐳 Add NSA implementations to `fla`. See kernels [here](https://github.com/fla-org/flash-linear-attention/blob/HEAD/fla/ops/nsa). - [2025-01] 🔥 We are migrating to `torchtitan`-based training framework. Check out the [flame](https://github.com/fla-org/flame) repo for more details. - [2025-01] 🦅 Add RWKV7 implementations (both kernels and models) to `fla`. - [2024-12] Add `flash-bidirectional-attention` to `fla-org` ([repo](https://github.com/fla-org/flash-bidirectional-linear-attention)). - [2024-12] 🎉 Add Gated DeltaNet implementation to `fla` ([paper](https://arxiv.org/abs/2412.06464)). - [2024-12] 🚀 `fla` now officially supports kernels with variable-length inputs. - [2024-11] The inputs are now switched from head-first to seq-first format. - [2024-11] 💥 `fla` now provides a flexible way for training hybrid models. - [2024-10] 🔥 Announcing `flame`, a minimal and scalable framework for training `fla` models. Check out the details [here](https://github.com/fla-org/flame). - [2024-09] `fla` now includes a fused linear and cross-entropy layer, significantly reducing memory usage during training. - [2024-09] 🎉 Add GSA implementation to `fla` ([paper](https://arxiv.org/abs/2409.07146)). - [2024-05] 🎉 Add DeltaNet implementation to `fla` ([paper](https://arxiv.org/abs/2102.11174)). - [2024-05] 💥 `fla` v0.1: a variety of subquadratic kernels/layers/models integrated (RetNet/GLA/Mamba/HGRN/HGRN2/RWKV6, etc., see [Models](#models)). - [2023-12] 💥 Launch `fla`, offering a collection of implementations for state-of-the-art linear attention models.Models
📈 近 7 天 Star 趋势
每日新增 Star（柱）与累计 Star（线）· 数据来自 GitHub
🏆 排名变化趋势
在各榜单中的排名历史（数字越小越靠前）
📋 历史上榜记录
每次快照的详细数据
| 日期 | 榜单 | 排名 | 周期新增 | 总 STAR |
|---|