Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
Critical systems research for ML infrastructure engineers optimizing LLM serving at scale; directly addresses the GPU-CPU memory bottleneck that limits practical deployment of sparse attention methods.
AI Summary
SPIN is a sparse-attention-aware framework that co-designs execution pipelines with hierarchical KV storage to overcome bottlenecks in long-context LLM serving. By unifying different sparsity granularities through a page-based abstraction, a locality-aware cache manager, and optimized metadata layout, SPIN achieves 1.66–5.66× higher throughput and 7–9× lower time-to-first- compared to vLLM, while reducing per-token generation latency by up to 58%.
Excerpt
Long-context LLM serving is bottlenecked by the cost of attending over ever-growing KV caches. Dynamic sparse attention promises relief by accessing only a small, query-dependent subset of the KV state per decoding step and extending the KV storage to CPU memory. In practice, however, these algorithmic savings rarely translate into end-to-end system-level gains because sparse methods typically operate at different granularities and thus rely on ad hoc, per-algorithm implementations. At the same