LKV Improves LLM Performance with Learned KV Cache Eviction

A new paper, "LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction," introduces a novel approach to improve the efficiency of large language models (LLMs) when processing long-context inputs ¹. The research, available on arXiv, focuses on optimizing the Key-Value (KV) cache, a crucial component in LLMs that stores information to facilitate efficient processing of text.

Long-context inference in LLMs is often limited by the linear growth of KV cache memory, which can become a bottleneck ¹. Existing methods for KV cache compression often rely on heuristics, which can lead to resource misallocation and suboptimal performance. LKV aims to address these limitations by formulating KV compression as an end-to-end differentiable optimization problem.

The LKV approach integrates two key components: LKV-H, which learns task-optimized global budgets, and LKV-T, which derives intrinsic KV importance without requiring the materialization of attention matrices ¹. This design allows LKV to bypass heuristic proxies and align compression directly with task objectives.

The researchers evaluated LKV on the LongBench and RULER benchmarks ¹. The results demonstrate that LKV achieves state-of-the-art performance, especially at high compression rates. For example, on LongBench, LKV achieved near-lossless performance while retaining only 15% of the KV cache.

The study's analysis highlights the importance of learned budgeting in achieving these results ¹. The data-driven allocation of resources, as implemented in LKV-H, proved essential in overcoming the limitations of hand-crafted heuristics.

The authors of the paper are Enshuai Zhou, Yifan Hao, Chao Wang, Rui Zhang, Di Huang, Jiaming Guo, Xing Hu, Zidong Du, Qi Guo, and Yunji Chen ¹. The paper was submitted on April 22, 2026.

The research suggests that learning head-wise budgets and token selection can significantly improve the efficiency and performance of LLMs, particularly when dealing with long-context inputs. The end-to-end learning approach allows for a more direct optimization of the KV cache, leading to better resource allocation and improved results compared to traditional heuristic methods ¹.

The development of LKV represents a step forward in optimizing LLMs for long-context tasks. By moving away from heuristic-based approaches and embracing a learned, data-driven methodology, the researchers have created a system that can more effectively manage the KV cache, leading to improved performance and efficiency ¹.

The paper's findings have implications for the development of future LLMs, suggesting that learned budgeting and token selection are critical for achieving optimal performance, especially as the context lengths of these models continue to increase ¹.