RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Eviction

¹Pusan National University · ²Sookmyung Women's University
{higok18, kbkong}@pusan.ac.kr · seungjun@sookmyung.ac.kr · ^†Corresponding authors

RestoreKV narrows the gap to full-cache behavior under aggressive compression at negligible one-time cost. Applied to KVzip+, RestoreKV reaches 86.4 RULER accuracy at 16× compression on the KVPress Benchmark, while adding negligible compression-time overhead.

Abstract

Query-agnostic KV cache eviction compresses a context once and reuses the resulting cache for arbitrary future queries, but performance can collapse under tight budgets. Existing methods primarily improve which original KV pairs are retained. We introduce RestoreKV, which complements this selection-based formulation with learned restoration under the same total KV budget. Our key insight is that, although the information lost through eviction is context-specific, the mechanism for generating its compact complement can be shared across contexts. After context prefill, a few restore tokens attend to the full KV cache in a single LoRA-adapted pass, generating a compact, context-conditioned restore cache. The base importance scorer and eviction rule remain unchanged, and the adapters are disabled for all subsequent queries and decoding. RestoreKV is trained through parameter-efficient self-distillation from the frozen full-cache model, optimizing only 0.4% of the parameters and requiring no task-specific tuning. Across four backbones and four long-context benchmarks, RestoreKV substantially reduces compression-induced degradation: on Qwen3-4B it improves 59 of 60 paired, budget-matched settings across five base eviction methods; at a 5% budget it raises KVzip from 38.2 to 73.2 on RULER-4K. Applied to KVzip+, RestoreKV reaches 86.4 RULER accuracy at 16× compression on the KVPress Benchmark, while adding less than 0.5% one-time cache-construction overhead in a 32K-context evaluation.

How RestoreKV works

A single LoRA-adapted restore pass, trained offline by self-distillation, generates a compact restore cache that is merged with the retained context under the same total KV budget.

Overview of RestoreKV. After context prefill, a single LoRA-adapted restore pass processes n restore tokens (n = 8 by default) with causal access to the full KV cache, generating context-conditioned KV pairs. These restore states are combined with the retained context states under the same total KV budget. LoRA is active only during restore-cache generation; subsequent query processing and decoding use the original frozen backbone. Training (offline): the restore-token embeddings and LoRA adapters (0.4% of a 4B backbone) are optimized by self-distillation from the frozen full-cache teacher.

Method

1) Restore-cache generation with LoRA. After the context is prefilled and the base method evicts KV pairs, a few learnable restore tokens attend to the full KV cache in a single LoRA-adapted forward pass, producing a small, context-conditioned restore cache that is concatenated with the retained pairs — all within the same total budget.

2) Self-distillation from the full cache. Only the restore-token embeddings and lightweight LoRA adapters are trained; the backbone and base evictor stay frozen. For the same context and query, the student using the budget-matched cache is optimized to match the frozen full-cache teacher's answer-token distributions — no task-specific tuning.

3) Inference. The restore cache is built once per new context. Adapters are then disabled, so every subsequent query and all decoding run on the original frozen model at the base method's query-time KV budget and cost. Component ablations show the gain comes primarily from attention-side adaptation (learned LoRA on q/k/v) rather than information memorized in the restore embeddings.

Consistent gains across backbones & benchmarks

Performance across four model backbones and four benchmarks. RestoreKV and RestoreKV+ are applied to KVzip and KVzip+ under the same KV budget. Both reduce compression-induced performance loss, with larger gains under tighter cache budgets.

Generalization across eviction methods (Qwen3-4B)

Method	RULER-4K			QASPER			QuALITY			LongHealth
Method	0.2	0.1	0.05	0.2	0.1	0.05	0.2	0.1	0.05	0.2	0.1	0.05
KVzip	91.4	80.1	38.2	38.2	33.9	18.0	66.0	58.5	46.3	71.5	70.0	61.0
+ RestoreKV	93.5+2.1	88.8+8.7	73.2+35.0	39.0+0.8	37.7+3.8	28.7+10.7	67.0+1.0	63.3+4.8	52.4+6.1	75.8+4.3	73.8+3.8	67.5+6.5
KVzip+	91.3	84.8	51.6	38.1	32.4	23.3	66.5	60.0	51.9	73.2	67.8	63.2
+ RestoreKV	93.3+2.0	89.7+4.9	70.7+19.1	40.9+2.8	38.5+6.1	31.6+8.3	66.8+0.3	62.9+2.9	57.9+6.0	74.2+1.0	73.0+5.2	71.5+8.3
ContrastKV	91.6	80.7	38.0	40.1	31.9	17.8	66.2	62.2	50.7	74.2	69.0	49.0
+ RestoreKV	92.3+0.7	84.3+3.6	40.2+2.2	40.9+0.8	36.0+4.1	21.6+3.8	67.6+1.4	63.3+1.1	51.9+1.2	75.2+1.0	70.5+1.5	54.0+5.0
SnapKV	33.8	20.6	12.7	22.4	19.0	14.3	53.9	53.0	42.5	41.0	36.8	36.2
+ RestoreKV	37.7+3.9	26.3+5.7	14.3+1.6	29.9+7.5	24.0+5.0	20.7+6.4	59.4+5.5	51.8−1.2	49.9+7.4	51.8+10.8	41.8+5.0	38.5+2.3
H₂O	8.0	3.5	3.2	22.0	13.8	14.4	59.2	50.4	41.5	60.2	48.8	37.0
+ RestoreKV	17.3+9.3	11.6+8.1	5.7+2.5	30.4+8.4	24.7+10.9	21.3+6.9	60.9+1.7	54.9+4.5	47.7+6.2	64.8+4.6	59.0+10.2	46.8+9.8

Columns are the KV budget ratio r (fraction of the full cache retained). RestoreKV is a budget-matched plug-in — base and + RestoreKV use the same total budget. Colored subscripts show the absolute change from the corresponding baseline; gains are typically largest at the tightest budget (r = 0.05).

What the restore pass recovers

Layer-wise context-attention recovery on RULER-4K (Qwen3-4B, r = 0.05). KVzip substantially suppresses attention from the final query token to non-sink context KV pairs. RestoreKV recovers part of the lost attention (shaded), raising the average context-attention mass from 0.56% to 1.30% and reducing KL divergence from the full-cache model by 16% (0.198 → 0.167).

Negligible one-time cost

Efficiency analysis (Llama-3.1-8B-Instruct, 32K context). (a) Restore-cache generation adds only ~0.04 s (0.5%) and 84 MB (0.4%) on top of any base scorer. (b) Attention latency and KV memory are nearly identical to the base method — the query-time budget and decoding cost are unchanged.