Pytorch for Beginners #30 | Transformer Model - Position Embeddings
In this tutorial, we’ll learn about position embedding, another very important component in Transformer Layer. We’ll first try to understand why we need it in a transformer model and look at some basic approaches and their limitations. Next, we’ll discuss the approach proposed in the paper, and try to elaborate how it solves the challenges raised in the basic approaches. Also, we’ll look at why we need multiple frequencies with both sine and cosine to generate the position embeddings. At the end we’ll also learn the reasoning behind summing the word embedding with position embedding instead of concatenation.
In the next tutorial, we’ll implement and visualize to make our understanding of position embedding more solid. Stay tuned!!
#pytorch #tutorials #transformer #position #embedding
