Editing Positional Encoding (section)

== Positional Encoding Mechanisms ==
=== Sinusoidal Positional Encoding ===
There are different ways of implementing positional encoding.  The original approach was the sinusoidal positional encoding proposed by Vaswani et al. in [[Attention is All You Need]] (2017). This method uses sinusoidal functions to encode the position of each token as a vector of the same dimension as the token embedding. The formula for computing the positional encoding vector for a given position and dimension is:

<math>PE(pos, 2i) = sin(pos / 10000^{2i / d_{model}})</math>

<math>PE(pos, 2i+1) = cos(pos / 10000^{2i / d_{model}})</math>

where <math>pos</math> is the position of the token in the sequence, <math>i</math> is the index of the element within the vector, and <math>d_{model}</math> is the size of the token embedding. The intuition behind this formula is that it creates a unique and periodic representation for each position, where different dimensions have different frequencies. This allows the model to capture both absolute and relative positions of the tokens, as well as generalise to longer sequences than seen during training.

The sinusoidal positional encoding is applied to the transformer model by adding it to the token embedding before feeding it to the encoder or decoder. The positional encoding vector has the same size as the token embedding vector, so that they can be summed element-wise. The resulting vector contains both semantic and positional information of the token, which can be processed by the attention and feed-forward layers of the transformer.

=== Rotary Positional Embedding (RoPE) ===
Rotary Position Embedding (RoPE) is a method for encoding positional information in transformer models proposed in the paper [[RoFormer: Enhanced Transformer with Rotary Position Embedding]]. 

Prior position encoding mechanisms like sinusoidal position encoding used in the original Transformer model directly added position embeddings to the input token embeddings. This can make the model less flexible when dealing with variable sequence lengths.

Unlike sinusoidal position encoding RoPE encodes absolute position information using rotation matrices rather than adding position embeddings. The query, key, and value vectors in self-attention are multiplied by these rotation matrices before self-attention is applied. 

More specifically, the input token embeddings are rotated by angles that depend on their position in the sequence. The rotation angles are predetermined constants based on the dimensionality of the embeddings. This rotation encodes relative position information between tokens.

Some key advantages of RoPE:
* It naturally incorporates relative position information through the rotation operations. The inner product between query and key vectors will decay with increasing relative distance between tokens, which matches the intuition that distant tokens should have less connection.
* It keeps the norm of the token embeddings unchanged, so it can be flexibly combined with linear attention mechanisms like Performer.
* There are no restrictions on maximum sequence length like with learned absolute position embeddings.
* It showed improved performance over baseline Transformer models across machine translation, language modeling, and GLUE benchmark tasks.

RoPE embeddings have opened up the potential for increasing [[Context window|context lengths]] by the application of [[RoPE Scaling]].