MIT Researchers Propose Cross-Layer Attention (CLA): A Modification to the Transformer Architecture that Reduces the Size of the Key-Value KV Cache by Sharing KV Activations Across Layers
The memory footprint of the key-value (KV) cache can be a bottleneck when serving large language models (LLMs), as it ...
