Editing Attention is All You Need (section)

== Novel Mechanisms ==
The paper introduced two types of attention mechanisms: scaled dot-product attention and multi-head attention. Scaled dot-product attention computes the weighted average of a set of values, where the weights are derived from the dot products of a query vector with a set of key vectors. The dot products are scaled by a factor of the square root of the dimensionality of the vectors to prevent large values from dominating the softmax function. Multi-head attention applies scaled dot-product attention multiple times in parallel, using different linear projections of the query, key, and value vectors. This allows the model to attend to different aspects or subspaces of the input and output sequences.

The paper also introduced two novel components: [[Positional Encoding|positional encoding]] and layer normalization. '''Positional encoding''' is a way of injecting information about the relative or absolute position of each token in the sequence into the model. The paper used sinusoidal functions to encode the position as a vector that can be added to the input embeddings. '''Layer normalization''' is a way of normalizing the inputs or outputs of each layer in the network. The paper applied layer normalization before each sub-layer (such as attention or feed-forward) and added a residual connection after each sub-layer.