DLite: History-free Speculative Decoding for Agentic AI
Presents novel inference optimization technique for researchers working on LLM acceleration and agentic systems.
AI Summary
DLite introduces history-free that achieves 2-6× acceptance length and outperforms DSpark by up to 70% for agentic AI workloads.
Excerpt
Speculative decoding has become a mainstream technique for accelerating large language model (LLM) inference by using a lightweight draft model to generate candidate tokens that are subsequently verified by the target model. However, agentic workloads introduce long and continuously evolving contexts and frequent multi-turn interactions. Under such workloads, we observe that existing draft models suffer from up to an 80% degradation in acceptance length, while maintaining their historical KV cac
