Accelerating Mixture of Block Attention via Dynamic Routing, KV-Centric Execution and Runtime Pipelining
Abstract
LightningMoBA combines dynamic routing, KV-centric ownership, and runtime pipelining to accelerate long-context attention while maintaining or improving quality.
Mixture of Block Attention (MoBA) reduces long-context computation by selecting key-value (KV) blocks for each query. FlashMoBA implements MoBA efficiently but retains a fixed Top-k budget and query-centric schedule. Fixed Top-k applies one budget across different score distributions and couples routing to task granularity; query-range ownership fragments K/V reuse when several ranges select the same column. We present LightningMoBA, which co-designs selection and execution through dynamic routing, KV-centric ownership, and runtime pipelining. Dynamic routing thresholds MoBA's per-query block scores and directly emits variable-cardinality sparse metadata. KV-centric ownership groups selected rows by (query head, KV block) column, splits high-fanout columns into bounded tasks, and reuses each task's loaded KV tile. On Hopper, pipelining overlaps regular K/V transfers and irregular query gathers with Tensor Core computation. Together, these mechanisms decouple retention from GPU scheduling granularity. At matched Top-16 retention across five checkpoints, LightningMoBA accelerates end-to-end generation by 1.20 to 1.60x while raising mean RULER score from 74.42 to 75.28.
Community
Sparse attention improves efficiency by reducing unnecessary computation. On GPUs, however, sparse-index construction, irregular memory access, and task scheduling can offset part of those savings. MoBA reduces computation by routing each query to relevant KV blocks, while FlashMoBA further optimizes GPU execution. Building on this foundation, we address two limitations: fixed Top-k budgets do not adapt to different query score distributions, and query-centric task ownership can lead to repeated KV-block loads across tasks, limiting data reuse.
We designed LightningMoBA to jointly optimize routing and GPU kernel execution, with three main improvements over FlashMoBA:
๐น Dynamic routing:โ Replace fixed Top-k with a relative threshold, allowing each query to retain a different number of relevant blocks. Sparse indices are generated directly during scoring, without maintaining an ordered Top-k candidate set or training an additional indexer.
๐น KV reuse:โ Within each attention head, group queries that access the same KV block so that each task reuses its loaded K/V. Work for heavily accessed blocks is split across multiple tasks to bound the per-task workload.
๐น Pipelined execution:โ On Hopper GPUs, overlap data movement and query gathers with Tensor Core computation to reduce stalls.
๐ Our experiments evaluate both end-to-end performance and the contributions of individual optimizations. Across five models and context lengths from 4K to 128K on the full RULER suite, matching the average retention budget to Top-16 yields 1.20โ1.60ร average end-to-end speedups per model over FlashMoBA, while the overall mean RULER score increases from 74.42 to 75.28. To separate routing gains from execution-backend improvements, we also compare both routing strategies under the same optimized backend. Across six configurations on two models, dynamic routing at the Top-64 budget improves both speed and quality, achieving 1.16โ1.56ร end-to-end speedups and 0.85โ5.95 percentage-point gains in RULER scores.
We also extend LightningMoBA to training. In a 100-step Qwen3-4B training experiment with 128K context, both routing strategies use the same optimized forward and adapted backward implementations. Dynamic routing achieves 1.23ร and 1.60ร end-to-end training speedups at Top-32 and Top-64, respectively, while training cross-entropy remains close between the two strategies.
We hope LightningMoBA can further improve MoBAโs training and inference efficiency and provide useful guidance for implementing other dynamic sparse attention methods on GPUs.
Get this paper in your agent:
hf papers read 2611.26268 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper