The KV cache is what takes up the bulk of the GPU memory during inference for large language models like GPT-4. Learn about how the KV cache works in this video!
0:00 - Introduction
1:15 - Review of self-attention
4:07 - How the KV cache works
5:55 - Memory usage and example
Further reading:
* Speeding up the GPT - KV cache ([ Ссылка ])
* Transformer Inference Arithmetic ([ Ссылка ])
* Efficiently Scaling Transformer Inference ([ Ссылка ])
![](https://s2.save4k.ru/pic/80bIUggRJf4/maxresdefault.jpg)