Recent ground-breaking research by Sepp Hochreiter, John Hopfield, and Dmitry Krotov reignited the interest in modern Hopfield networks and associative memories. The paper “Hopfield Networks is All You Need” by Sepp Hochreiter’s team introduced a modern continuous modern Hopfield network with exponential storage capacity and very fast convergence. The authors showed that the update rule of the new Hopfield network is equivalent to the attention mechanism of the popular successful Transformer architecture. John Hopfield and Dmitry Krotov then showed in their work on “Large Associative Memory Problem in Neurobiology and Machine Learning” that dense associative memories and modern Hopfield networks can be linked to models with biologically plausible pair-wise interactions between neurons. Together, these recent hallmark studies are important for understanding Transformers, improving attention mechanisms, while providing a link of these advanced architectures to biologically plausible neural networks. We are looking forward to these pioneers of AI exploring the implications of their findings from a theoretical, a biological, and a physics perspectives, and a discussion of future directions of research.
_____________________________________
IARAI - Institute of Advanced Research in Artificial Intelligence
www.iarai.org
